Back to news & views

Big Data – Deliver on a vision not on a compulsion

A brief glance at the Big Data Landscape from Dave Feinleib strongly supports the notion that the Big Data era is here. With ‘Analytics and Business Intelligence’ now topping Gartner’s list of CIO spending priorities, up from fifth in 2011, it seems as though more and more CIOs will be considering how they could use Big Data as part of their overall analytics approach.

But how to make best use of that investment without simply jumping on the bandwagon? Here are some areas you might consider:

Look to address a specific business challenge rather than just implement a Big Datasystem. The Big Data hype promises to unlock hidden relationships within your existing data – patterns that are just waiting to be discovered. While Big Data tools are powerful, they are only suited to certain types of problem, and the relationships they uncover may be meaningless to the business. A preferable approach is to set out to address a business challenge to focus your initial efforts and demonstrate tangible relevance to the business. On reflection, you may even find your current systems are capable of providing the information you need.

Understand where Big Data tools excel and where existing BI solutions are better. Allowing users to analyse clean, structured data interactively in real-time is still very much in the BI domain – and recent implementations are capable of significant scaling too. Big Data tools come into their own however for investigating unstructured or frequently changing information, or for detecting patterns in vast amounts of data across multiple dimensions. But not all businesses have these demands.

Ensure you have the appropriate skills and team organisation. A good team knits together an understanding of the business, data analysis skills and platform knowledge. Having all three skillsets working together is a powerful way to share understanding, allowing you to ask pertinent business questions, interpret the data correctly (and, importantly, not come to the wrong conclusions), and support and evolve the solution as your experience grows.

Think twice about keeping all the data you can, just because it might be useful one day. Many Big Data solutions trumpet their use of cheap commodity disk allowing you to retain granular data in an easily accessible online store, without worrying about throwing away information that might be useful someday. While this sounds attractive, there is still an underlying cost to this hoarding. Ensure you can justify the data you store based on your Big Data strategy.

Think wider than Hadoop. While it is a powerful platform, Hadoop (Apache’s popular open-source software for reliable, scalable, distributed computing) is not a panacea, and it normally requires a number of surrounding tools to make a useable Big Data solution. Other similar systems do exist and may be more aligned to your requirement. At least initially, simpler data analysis and visualisation platforms may be more appropriate to help you develop your approach to Big Data.

The bottom line – Big Data is moving into the mainstream and there are compelling reasons why CIOs of companies other than internet giants should start to look at these solutions. But a little upfront consideration means you can deliver on a vision, rather than on a compulsion.