NZ businesses struggle with data quality for AI adoption

New research from Datacom has revealed that New Zealand businesses may not possess the quality of data needed to fully leverage the potential of Artificial Intelligence (AI) and advanced decision-making tools. Only a mere 9% of companies in the nation regard their data as completely clean, while 30% believe that half or less of their data is free of issues.

Close to 47% of businesses estimate that about 75% of their data is clean. The term “dirty data” refers to data that is marred by issues such as duplications, becoming outdated, being insecure, incomplete, inaccurate, or inconsistent. Justin Gray, Managing Director of Datacom New Zealand, warns that this dirty data represents significant risks for firms especially as the use of AI becomes more prevalent.

Gray emphasised, “The quality of AI outputs is wholly dependent on the quality of data that is available – garbage in, garbage out. AI can be a powerful predictive tool and can identify patterns and trends within your business to guide strategic decisions, but if the data is off, then the AI analytics will be too.”

The survey also highlighted a disparity between businesses' reliance on data and their data hygiene practices. While many companies endorse solid intentions for their data, basic management practices appear deficient. According to the report, 40% of the surveyed businesses confessed to not conducting regular data cleansing, and an additional 13% were unsure about their organisation's practices.

Of the businesses that do conduct data cleansing, 51% are unclear about the regularity of this process. Data audit and assurance protocols are similarly inadequate, with 46% lacking these measures and 11% unsure of their existence. Despite these deficiencies, businesses frequently employ their data for critical functions such as streamlining operations (80%), strategy and decision-making (79%), improving customer experience (77%), and refining processes (76%).

Commissioned by Datacom, the research involved a survey of 200 senior data and IT managers to assess New Zealand businesses' readiness to effectively utilise their data. Gray noted, “The results show a clear need for data and IT managers to reevaluate their current processes if they want to use their data as the foundation for quality real-time analytics and insights, and if they plan to introduce AI tools and platforms into their business that will rely on the integrity and accuracy of their data.”

Businesses identified several specific barriers to optimal data utilisation. These included incomplete data (51%), duplicate data (46%), inaccurate data (43%), and low-quality data (39%). Compliance or regulation issues (34%), privacy risks (30%), and security risks (22%) were also highlighted as significant concerns.

“These issues mean businesses are very vulnerable if they are relying on this data to help them make strategic decisions. If they can’t trust the data, they can’t trust the insights,” said Gray.

The research further examined data storage practices among businesses. Many utilise mixed storage locations, with the most common being on-premise servers (53%), private cloud (35%), New Zealand data centres (34%), public cloud (31%), and offshore data centres (15%).

Gray pointed out that as emerging technologies evolve, the rate of data production and collection will surge, necessitating comprehensive plans for data management, storage, and protection. “Those companies that are relying on on-premise servers, for example, need to prepare for the rapid data growth that will come with greater AI usage and look at introducing back-up options so they aren’t left scrambling,” he advised.

Gray suggested that options such as Datacom’s data centres provide a secure and scalable environment capable of supporting data sovereignty issues, allowing companies to keep their data within New Zealand for regulatory and security reasons.

The survey also revealed that while 91% of businesses believe their data and security protocols are adequate, only 60% have formal data governance policies managed by trained staff. The survey, administered by Curia Market Research in May 2024, involved senior decision-makers from organisations with over 100 employees in New Zealand.