With the volume of global data to grow to 163 zetabytes by 2025 (ten times the volume seen in 2016), the implementation of Big Data analytics solutions is becoming an increasingly critical aspect of the digital transformation journey.

This transformation represents both an opportunity and a challenge, and the promise of better and faster data-driven decision making has seen Big Data rise to the top of the forward-thinking business executive’s agenda.

For these high-ranking decision makers, the attraction is not just the access to information that such solutions provide. Indeed, the biggest attraction is the ability to analyse and act upon this information in a timely manner.

This ability not only creates competitive advantage in the marketplace, but also enables sustainable management of communities and natural resources, and promotes the appropriate delivery of social, health care, and educational services.

Such benefits have not escaped the attention of business executives here in the Middle East. And with board-level mandates for the implementation of digital transformation now commonplace across the region, we are seeing greater use of analytics solutions by organisations of all shapes as sizes.

Most transformed business processes are fundamentally reliant on analytics of some sort. And in some cases, this reliance stretches to the new breed of advanced and predictive analytics that draw on Big Data as their source.

However, despite the phenomenal growth being seen in the Big Data analytics space, it is still a relatively small part of the overall analytics market, which continues to be dominated by the more classic forms of business intelligence (BI) and business reporting.

That said, interest in advanced analytics is rapidly growing, particularly in relation to cognitive techniques such as machine learning. This trend is set to intensify over the coming years as we witness an explosion in the number of connected devices making up the Internet of Things (IoT), as this will inevitably produce large amounts of data that can be used to improve current processes and develop new business models.

The Big Data technology and services market saw some consolidation last year, and like its parent — the business analytics (BA) technology and services market — grew in maturity when compared with previous years.

Under such circumstances, one might expect suppliers to reduce their focus on the Big Data analytics space, or at least rein in their innovation. One might also expect to see a sudden stiffening of the barriers to entry in this market.

But none of this has happened. Indeed, we have seen more suppliers join the ranks of incumbents in introducing innovative (and disruptive) Big Data technologies and solutions to the market. And the breakneck speed of this innovation has seen the job market struggle to keep up.

The recent global survey of 367 data scientists and other knowledge workers involved in the advanced and predictive analytics market showed that 72 per cent of respondent organisations have a shortage of data scientists and 54 per cent are encountering difficulties when trying to hire new data scientists.

Attempts are being made to address this shortage through higher-education initiatives and on-the-job training. But, is simply having more data scientists really the answer to ensuring that organisations are able to derive greater value from analytics?

The results make it clear that data-driven decision making is not only about the number of data scientists or the implementation of better algorithms; these human resources alone will not suffice in creating and nurturing a truly data-driven decision-making culture.

Of course, having a sufficient number of high-quality analysts in place is critical to achieving such aims. But this must be supplemented by the right data and the right software, as well as by the provision of board-level support and investment and the elimination of friction in internal processes.

Only then can we begin to see the levels of collaboration between data scientists, line-of-business heads, and IT leaders that are required to ensure organisations derive the maximum value possible from their pursuit of Big Data analytics.

The columnist is group vice-president and regional managing director for the Middle East, Africa and Turkey at global ICT market intelligence and advisory firm International Data Corporation (IDC). He can be contacted via Twitter @JyotiIDC. Content for this week’s feature leverages global, regional, and local research studies undertaken by IDC.