Using analytics and big data to predict the future in your industry – The Bernard Marr column

Wouldn’t it be great to be able to predict the future? When it comes to making decisions, our success at pretty much anything depends on our ability to predict the future outcomes of our present-day actions.

If you had a crystal ball or a time machine, you could remove all of the guesswork. You’d directly see the products of your labour, and instantly know whether or not your present efforts were on the right track.

The crystal ball dilemma

Starting a new diet or exercise regime? Good for you! It’s almost certainly going to have a positive impact on your life. But – is it the best one? Is the time, effort and money you’re putting into it being used at optimal efficiency? If you could see into the future, you’d be able to tell how it would work out, then modify your behaviour in the present day accordingly.

In business, the best stand-in we have for our frustrating lack of crystal balls and tardiness are trends. Whether we intend to buck them or follow them, we seek to identify common threads of activity in industrial practices, customer behaviour and anything that could make a difference to our bottom line.

Predicting future trends

Spotting and monitoring behaviours and patterns allows us to take a stab at predicting where things are heading, how demand for our products or services will change over time, and what will prompt that change. 

What can we do to respond to change when it comes, to increase demand when it’s low, and ensure supply when it’s high? These are all important considerations.

In recent years, the application of predictive analytics, combined with the vast, ever-growing and increasingly varied amount of information we collect and store, is taking a lot of the guesswork out of that process.

Predictive power of big data

Big data is the name that has been given to that combination of advanced computer analysis and huge datasets – although I prefer the term ’smart data’ as it emphasises that the analysis is just as, if not more, important than the size of the data. And by cleverly applying it to spot emerging trends we have the closest thing yet to a crystal ball.

It makes sense, if you think about it. Until recently, trend analysis and prediction often came down to ‘gut instinct’ – that feeling that people who are confident in their abilities get, and which they often feel puts them at the top of their game or gives them an edge over their competitors.

A gut feeling is not a good predictor

I’m sorry to have to break it to you if you’re one of those people – the reality is, gut instinct often comes down to pure luck.

Throughout history, and particularly in business, plenty of ‘great’ strategists have fallen flat on their faces because their gut instinct let them down at a crucial point. Experience certainly helps, but experienced people can be unlucky just as often as rookies – if by 'luck' we mean anything outside of our control which could interfere with our desired outcome.

Data-driven predictive analysis

What data-driven predictive analysis does is remove the egotistical hubris of the ‘gut feeling’. It also minimises the random influence of luck, by making us aware of as many of the factors which could influence our outcome as possible – giving us the vital opportunity to bring them under our control.

Marketing is a great example. Times have changed since George Gallup introduced market research by polling customers in the 1930s. With the advent of social media and the internet, we’re used to (knowingly or unknowingly) sharing vast amounts of data about ourselves, our interests, habits, likes and dislikes – and savvy marketers have been quick to tap into this.

Trending topics flash across Facebook and Twitter every day, making it easier than it has ever been before to work out what people are looking for and what they want. Products and services can then be marketed to fill those needs. Services such as Trendera (external link) and Trend Hunter (external link) collate this data and use it to answer specific questions for their business customers.

Spotting trends

The ability to know what the public wants before they know it themselves is every marketer’s holy grail.

In retail, online and offline customer behaviour can be measured to microscopic detail. That data can be compared with external data, such as the time of the year, economic conditions and even the weather, to build up a detailed picture of what we’re likely to buy, and when.

Walmart, a business which has put the big data ethos at the heart of its operations, bases all of its stocking decisions on data algorithms. Thanks to a famous early experiment in big data towards the start of the last decade, they learned that demand for beer and pop-tarts would often increase in areas forecast to be hit by severe weather such as hurricanes.

Since then, they’ve collected and drilled into data with ever-increasing vigour, yielding more and more insights which inform their stocking and logistical operations year in, year out.

The basic principles of data-driven predictive analytics are being used to spot trends in all industries. The fortunes of internationally-renowned fashion and lifestyle brands are made or broken by their ability to predict what we want. In the not-so-old days, this would often involve identifying ‘trend setters’ – those ultra-cool individuals who the rest of us mere mortals aspire to be like.

Industry ‘spotters’ would pick out these cool kids at the clubs and boutiques and take notes on what they wore and what products they used, to help them draw up their own picture of what was set to be hot or not.

These days, there’s no need to comb the streets. Millions of photos are uploaded to Facebook, Twitter and Instagram every day, and algorithms have been developed to scan them as quickly as they’re shared, picking out the brands of drinks we’re enjoying in our holiday snaps, and even our moods.

People with large numbers of friends or followers can generally be assumed to be more sociable and influential, so the algorithm can identify them as more valuable trend indicators.

Spotting trends in small businesses

Today, our ability to predict trends is limited only by our ability to think SMART (external link) about the data and the technology we have available to us, rather than limited by the data and technology itself.

Despite the name big data, it’s no longer something which is solely for big business, with dedicated IT teams and warehouses stuffed with years of collated records. Much of the technology is based on free, open-source software, or inexpensive, software-as-service cloud-based solutions.

What's more, a lot of businesses have gained very valuable insights from the free, huge public datasets made available by companies like Google (Google Trends is a fantastic tool) and government services such as data.gov.uk.

This makes it perfectly viable for many small and medium-sized enterprises to have a crack at predicting the future themselves.  We might not yet have the technology to actually see into the future, but now we have the ability to remove the guesswork and replace it with cold, hard data-driven insights. In my experience, the best predictions occur when we combine those hard facts with solid industry experience.

For further insights, read Bernard’s previous column on how small businesses can get started with big data.

For information on data insurance or to get a quote for Hiscox policy, visit our cyber and data insurance page. Our cover provides protection for businesses in the event of a data breach or cyber attack.

Have you managed to predict any trends using data? We’d love to hear about your experiences, so fire away in the comments below.