Stop trying to tame the data beast and learn to herd data critters
“The world’s most valuable resource is no longer oil, but data.” – The Economist, 2017
“Data is crucial to the way government delivers services for citizens, improves its own systems and processes, and makes decisions.” – NAO, 2019
The pressure is on for many organisations to be ‘seen to be using big data’. Designing and implementing big data programmes is a unique challenge.
It is not about technology
The difference now is that the challenge has evolved from gripes around just technology and integration (hands up if you’ve built a successful Hadoop data ‘lake’) to also encompass people and ethics. We are told in the media that business decisions are increasingly more data-driven, but our ability to locate increasingly smaller needles in increasingly bigger haystacks is still limited by the abilities of employees as analysts to leverage information, and them being able to navigate the growing importance for responsible data handling and access. Companies have realised that metadata, lineage and data governance help address this; but have also realised that (despite the promises of AI and data science) making effective and appropriate use of big data is still quite a lot of hard work, and costly. Speed of interpretation and inference are everything; the job of the analyst and business decision makers to deliver insight using big data solutions is only getting harder.
There is a need to grow in-house core data capabilities
What does this mean for enterprise? At DMW, we are seeing a growing trend in clients wanting to take ownership back of analytics, data capabilities and data stores from outsourced suppliers and System Integrators, in order to bring their staff closer to information, understand its value, and improve capability in analytical decision making. This coupled with convergence of data standards, improved ease of integration and adoption of open source technologies through cloud providers means businesses are more open to self-sufficiency … but struggle with understanding how to achieve it.
Empowering your people to try and test ideas
With maturing of cloud, ephemeral platforms, automation and agile and DevOps practice, we believe that the industry is at a tipping point, and learning to herd your data critters, and accepting that it is pointless (and impossible) to tame the data beast, is key. The days of running a costly big data and analytics estate are gone. To succeed in data, businesses must empower their staff to break down the data sources and the problem; and leverage technology that no longer needs to differentiate data as ‘big’. This means changing skills, knowledge and ways of working, and investing in communities and a learning organisation in data and analytics practice that don’t just straddle business and technology; but blur their ongoing interaction. Orienting around blended data and analytics expertise and multi-disciplinary teams responsible for not just delivering, but qualifying value, maintaining delivery standards and contributing back to communities allows for the shepherding of knowledge. Unicorns don’t exist, but by enabling staff to test ideas and leverage solutions they can grow sustainably themselves, you can produce plenty of horses.
DMW has significant experience in delivering successful big data projects within the public and private sectors and enabling analytics capability.