Do You Have Too Much Faith in ‘BIG DATA’ ?
It’s well documented in behavioral research that faith – in anything – can blind (and bind) us from seeing the shortcomings of whatever it is we have faith in. So it is with the ubiquitous crutch of big business today – ‘BIG DATA.’
Ubiquitous because there’s enough digital exhaust out there to cover every inch of the planet in 3 feet of smog. Enough for nearly every topic you can come up with, provided you don’t look to close at the quality of it. Analysis and infographics at the click of a dashboard button makes it addictive to keep mining for something that makes the marketer’s heart beat a little faster.
Having faith in ‘BIG DATA’ means you don’t stop to question the implicit assumptions you make every time you rely on it. When I ask believers what makes them believe, I typically hear one or more of these:
Internet data is ‘pure data’
Algorithms don’t involve people making decisions so they’re not biased
It’s cheap and fast (‘internet speed’)
Everyone is doing it – so we have to
Our CEO/CMO/COO etc. loves big data
It keeps us from missing out on anything
Where did these assumptions come from? Anyone connected to ‘the internet’ has been exposed to years of propaganda from those friendly five oligarchs of Silicon Valley whose lifeblood of advertising depends on faith in the all-knowing power of ‘BIG DATA.’ Now that it’s a research norm, who would challenge the word of the five biggest firms in the world?
I’m not here to challenge anyone’s faith, only to challenge a few assumptions that are overlooked in the zeal to unleash the algorithms:
One, Internet data isn’t pure – many datasets are aggregations across any number of sources and dimensions, with questionable heritage (‘proprietary’).
There’s cheap, fast or good – in a pick any two scenario, good seems left out.
Algorithms are programs of decisions made by humans – not aliens, and so contain biases. As for missing out, surely if you’re looking in the same places that everyone else is, you’ll very likely find the same things.
The most dangerous assumption is that this kind of data can help you decode human behavior in the real world. If you are using ‘BIG DATA’ for that, there’s a big problem. Humans make a majority of their decisions based on intuition/emotion and rationalize them afterwards, and the context within which the decision takes place has a great deal to do with it. ‘BIG DATA’ is missing both the emotion (spoiler alert: emotion is analog) and the context – which can make for some nasty surprises.
This handicap can be overcome by adding some ‘small’ data to the mix – proper ethnography, depth interviews, well-moderated focus groups. So here’s my modest proposal: for better insights and fewer nasty surprises, use the ‘BIG DATA’ for direction or pattern recognition, then rely on the small data to help decode contextual behaviors and emotion.