AI infrastructure divide defines generative AI success – Highlights from VotE: AI & Machine Learning

Introduction

AI workload demands continue to expand, with generative AI a driver of further investment in infrastructure. As organizations shift from experimenting with generative AI capabilities toward operationalizing them, new pressures are emerging on an already stretched infrastructure setup at many organizations. 451 Research's Voice of the Enterprise: AI & Machine Learning, Infrastructure 2024 study, a survey of more than 700 mid-level and senior IT and line-of-business professionals, provides a detailed view into the strategies organizations are engaging in to meet this pressure. The study explores AI execution venues, tools, purchasing strategies and challenges.

The Take

The opportunity for generative AI is well acknowledged. Almost all organizations perceive staff to be using generative AI capabilities, although in many instances the tools being used are not sanctioned or integrated offerings. Few organizations are in a position where experimentation with generative AI has led to embedded and broadly deployed capabilities. Considering the security and privacy risks of staff using tools without adequate levels of observation or controls, the pressure for organizations to rapidly deploy capabilities is sizeable. This pressure is further exacerbated by challenges of data theft and inaccurate responses associated with generative AI, and the limited value that can be achieved with generations that are not suitably tailored or anchored in enterprise data. As the dust starts to settle on the generative AI explosion, a small cohort of generative AI leaders appears to be leading the pack. These leaders have invested in a more robust AI infrastructure setup.

Summary of findings

AI workload and data requirements are on the rise. The vast majority of respondents (83%) forecast that AI workload demands would increase this year. Unsurprisingly, this increase is echoed by the majority of respondents who predicted that data needed to train models would increase by at least 25% in the next 12 months. Fourteen percent of respondents expect data needs would increase by more than 50%. Notably, the data assets organizations are set to employ are increasingly diverse — with significant and growing interest in leveraging unstructured rich media and textual data. Rearchitecting to account for rising demand for data, and the storage and effective use of new data types, will likely be a challenge for many organizations.

Generative AI is a driver of this multimedia trend. Developments in areas such as image and video generation, and a desire for tailored outputs, will likely encourage companies to rethink their storage strategies. "Data types" and "data volumes" are two of the top four drivers of organizations employing a distinct strategy for generative AI. These organizations, which represent the majority of our respondent base, also see the cost and security implications of the technology trend as major considerations. Investment in generative AI also appears to be having an impact on confidence that organizations have the infrastructure in place to meet AI workload demand. The proportion of respondents suggesting their IT environment was able to meet future AI demands without upgrades falls to a low of 29%.

Most organizations are still experimenting with generative AI. Few organizations are equipped to take full advantage of generative AI. While 73% of respondents are from organizations that have some level of investment in generative AI, the vast majority are either still trialing capabilities or have a limited rollout isolated to a team or project. Just 18% of respondents are from organizations that have integrated generative AI across the business, and these organizations represent a quite different profile. Respondents from organizations that have rapidly accelerated down the generative AI road map are far more confident that their organization's IT environment is able to consistently meet ML workload demands. These organizations make use of a wider range of execution venues for data preparation, model training and inference. In addition, respondents from these organizations are less likely to be concerned by perceived security or reliability challenges. Compared with organizations that have more limited deployments, generative AI leaders are also less likely to have a shortage of graphics processing units or storage capacity.

AI road maps are expensive to deliver. Budget has emerged as the number one driver of project failure as organizations struggle with the costs of AI initiatives. More than one-third of respondents (35%) suggest that budget considerations led to AI project abandonment in the past 12 months — a significantly higher proportion than longstanding areas of challenge like executive support (21%) or inadequate tooling (19%). After security, cost is the largest factor that influences AI infrastructure decision-making.

Trying to rein in AI infrastructure expenditure will prove challenging in the context of the broad upgrades enterprises have planned. More than one-third of respondents perceive infrastructure limitations as preventing their organizations keeping pace with AI advancements (37%), and a similar proportion see infrastructure limitations as a restraint on model performance (36%). Accordingly, spending on computing devices, AI accelerators, storage, networking and CPU resources is forecast to increase markedly within the next 12 months, and few organizations (11%) predict overall spending will do anything other than increase. For 10% of respondents, organizational spending on AI infrastructure is forecast to increase by more than 50%.

The growing importance of AI has remodeled infrastructure purchasing practices and priorities. Growing spending has elevated the importance of AI infrastructure decision-making within organizations. Paired with the broad applicability of AI across the business, this trend has contributed to a greater range of stakeholders being brought into purchasing decisions. Operations, finance and information security management roles are emerging as infrastructure influencers, with representatives of these functions being brought into purchasing decisions, while rarely being the primary decision-maker.

Sustainability appears to be taking a backseat, as the perceived importance of AI has led to organizations feeling a need to build out infrastructure quickly that meets security, cost and reliability requirements. Sustainability, while still seen as a key factor by 33% of respondents, is also seen as less critical than data privacy, flexibility and scalability requirements. The proportion of organizations that have invested in public cloud for AI workloads that suggest their decision-making around region or availability zone was a primary decision factor fell significantly year-over-year. In 2023, the majority of respondents said sustainability was a factor in region or availability zone selection (64%), and they made a decision because of it. This figure falls to just 48% this year.