Research into data and complexity reveals immense pressure on companies

The insurance industry knows well the organizational challenges related to the exponential growth in data volumes and rapid innovation across the data stack. New research from Boston Consulting Group (BCG) explores this challenge in a survey of 300 global organizations. Titled, A New Architecture to Manage Data Costs and Complexity, the report highlights three main trends shaping data advancements today.

First, they say the volume and velocity of data are increasing. Data use cases becoming more accessible has resulted in the growth of “citizen data scientists,” but the use cases are also becoming more specialized and there is a gap between the requisite and available skill sets. They also explore how technology advancements are shifting pricing.

“The research indicates that organizations are facing a perfect storm of cost, complexity and talent when it comes to supporting critical analytics and artificial intelligence (AI) initiatives,” Starburst Data announced in a statement about the research. They add that more than 50 per cent of data leaders say architectural complexity is a significant pain point. Vendor proliferation is also a major issue. “For larger companies with a more mature data stack, the total number of unique data vendors has nearly tripled in the last decade – up from about 50 to over 150 today,” they write.

“The gap between more advanced analytics use cases and technologies, and the analytics skill sets required, is currently limiting the business outcomes AI can drive. Only 54 per cent of managers believe that their company’s AI initiatives create tangible business value.” 

Suggesting that more organizations today are reaching a breaking point, the report warns that the combination of complexity and costs could cripple operations for many companies. “As companies move up the maturity curve, from data-driven to AI-driven organizations, architectural complexity and fragmentation inevitably rise,” the report states. It also goes on to discuss lessons learned. Among those lessons discussed, they say architecture will become more decoupled, federated and service-oriented, new standards and protocols will emerge. They also warn that the need to manage spiraling costs may drive many enterprise data architecture choices.

“Stay alert for new data market categories, competition and tools,” the report states. “Participate in establishing new standards.” Finally, they conclude by encouraging firms to “meet customers where they are and help them with change management.”