A primer on AI governance

In 2020, concerns started to escalate about the ethical and responsible use of artificial intelligence as it became increasingly deployed to support business use cases, creating the potential for bias, discrimination, data privacy issues and other problems. AI governance emerged as a discipline to enable organizations to adopt AI in an ethical and responsible way. In this report, we aim to define AI governance and shed light on the evolving vendor landscape for AI governance platforms.

The emergence of AI governance platforms has helped make AI governance less nebulous by grounding it in a workflow supported by a set of features. Capabilities such as model risk scores and assessments, documentation of AI processes, model explainability, the ability to document evidence of compliance, and the translation of regulations into codified procedures to meet them all bring structure and context to AI governance. Furthermore, the EU AI Act is bringing clarity — as well as urgency — to AI governance by defining what is acceptable and what is not, as well as the financial consequences of noncompliance.

In North America, there seems to be a groundswell of positive sentiment from the public for federal regulation of AI, which could influence the introduction of regulation in the US and act as a fresh driver of AI governance. In 451 Research's population-representative VoCUL: Connected Customer, Trust & Privacy 2024 survey, 65% of respondents indicated that they would be likely to support federal regulation of AI in the US. However, all regulations will need to keep up with the furious pace of AI development to be effective. That arguably will be AI governance's biggest challenge in the future.

Early definition

AI governance has evolved since it emerged, which is a key reason why it is not well understood. Four years ago, AI governance referred to the need to build and deploy AI systems that were managed to maximize benefits and prevent harm. The emphasis was on ensuring that AI was in line with societal values and respected human rights so that AI-influenced decisions, for example, were not biased or discriminatory. However, there were very few AI regulations in place — and AI governance was frequently perceived as "nice to have" rather than critical.

AI governance 2.0

AI governance undoubtedly still has an ethical and responsible AI mandate. This is a major reason organizations have established advisory boards to provide thought leadership and guidance on how to identify and mitigate risks from AI products developed in-house or by third parties. Indeed, 41.6% of respondents to our Voice of the Enterprise: Data & Analytics, Data Architecture for AI 2024 survey said their company has a corporate AI ethics board, illustrating that AI governance culture is taking root among organizations. An AI governance culture is critical for risk mitigation as well as building trust in AI — and therefore, now fundamental to its meaning.

That said, AI governance is also increasingly about providing documentation and other supporting evidence that a business's AI models adhere to regulations, standards and internal compliance practices. The EU AI Act, which was established in early August and will be enforced in stages over the next three years, is pivotal to the changing nature of AI governance. It is the first comprehensive regulatory framework aimed at governing AI and establishing rules based on risk levels associated with AI applications.

As such, the EU AI Act represents the first significant step toward formalizing AI governance and making it mandatory — as well as penalizable. However, it is worth noting that other countries are also now introducing AI regulations. For example, China's Interim AI Measures took effect in August, representing the country's first specific administration regulation for generative AI.

The EU AI Act has several levels of penalties if breached. Supplying incorrect, incomplete or misleading information to notified bodies and national component authorities can result in a fine of 1% of global revenue or €7 million ($7.7 million) under the EU AI Act. Engaging in prohibited AI practices — such as classifying people based on personal characteristics, socioeconomic status or behavior — will incur fines of up to €35 million, or 7% of a company's annual revenue. The penalties incurred are not only financial. Damage to an organization's reputation and brand will also likely be considerable.

Legislation is not the only factor changing the nature of AI governance. Barely a year ago, AI governance was concerned only with managing the risks associated with machine-learning models. Generative models have changed the remit of AI governance by introducing a series of new business risks — including copyright infringement, plagiarism and misinformation — that require mitigation or management.

Therefore, AI governance now involves the ethical and responsible adoption of generative AI as it is deployed for business use cases, as China's Interim AI Measures exemplify. Organizations also consider this challenging. Slightly more than one-quarter (28%) of respondents to our VotE: AI & Machine Learning, Use Cases 2024 survey said regulatory compliance was a key challenge to GenAI adoption within their organization.

AI governance vs. data governance for AI

Companies may also struggle to distinguish between data and AI governance. This challenge arises partly because both disciplines involve organizations' data assets, and because some data governance vendors have engaged in "AI governance washing."

Data governance is concerned with the policies, processes and standards that determine how an organization's data is handled throughout its life cycle so that it is accurate, consistent and compliant with regulations. That said, data governance can increasingly involve other data-related practices such as data security. As we have previously discussed, data governance, privacy and security initiatives are converging, creating complex interdependencies.

AI governance focuses on the governance of a company's AI models. While concerned with the data associated with those models, the primary focus is the models themselves. AI governance is concerned with how models are designed, developed, implemented and maintained to comply with external regulations and internal policies. AI governance requires continuous model monitoring and updating of policies to ensure that models adhere to them.

Equally important, AI governance necessitates making the innards of every AI model explainable, as well as documenting the evidence to all relevant stakeholders in a language they understand. An organization's compliance officer, or a regulator, will therefore require detailed model descriptions in business language, for example, not code.

AI governance platform landscape

AI governance platforms provide a range of capabilities to help support the lawful and ethical deployment of artificial intelligence. Core features include visualizations and explanations to enable the interpretability of models, documentation of AI processes for transparency and accountability, model fairness and bias detection, and continual model monitoring and auditing to ensure ongoing compliance with regulations. Other key features include risk management to assess and mitigate model risks, as well as customizable policies and controls tailored to specific sectors or regulatory requirements.

The AI governance platform category has so far been mainly occupied by specialist startups. AI governance pure plays include Credo.AI Corp.'s Credo AI; Monitaur Inc.'s Monitaur; 2021.ai ApS's 2021.AI; Fairly AI Inc.'s Fairly AI; Saidot Ltd.'s Saidot; Konfer; Holistic AI Ltd.'s Holistic AI; Fairnow Inc.'s Fairnow; and Enzai Technologies Ltd.'s Enzai. That said, established software vendors are now entering the space led by International Business Machines Corp., which released its AI governance offering in late 2023.

Furthermore, we expect new market entrants from complementary software sectors to move in to feed organizations' appetite for AI governance offerings. Slightly more than one-third (35.1%) of respondents to our VotE: AI & Machine Learning, Use Cases 2024 survey said their organization planned to invest in AI governance tools, platforms or functionality over the next 12 months.

Data science platform vendors are an example of an established category of players entering the AI governance realm. This is because AI governance complements existing capabilities already offered for machine learning operationalization and, increasingly, large language model operationalization. Dataiku Inc.'s Dataiku, Domino Data Lab Inc.'s Domino Data, H2O.ai Inc.'s H2O.ai, DataRobot Inc.'s DataRobot and SAS Institute Inc.'s SAS Institute could therefore become purveyors of a broad range of AI governance tools.

Cloud hyperscalers are also likely to address AI governance because it represents a logical extension to their legacy data science, machine learning, and GenAI development and deployment capabilities. Thus, Amazon.com Inc.'s AWS, Microsoft Corp. and Alphabet Inc.'s Google could become "go to" providers in the future as well.

Other potential market entrants include data platform developers. Cloudera Inc. and Databricks Inc., for example, already support data governance, data science and machine learning. The addition of AI governance functionality would reinforce their respective expertise for machine learning and GenAI model deployment. Vendors such as Palantir Technologies Inc. and C3.ai Inc. which focus on ModelOps as a core capability, could also make AI governance plays to double down on model operationalization.

This article was published by S&P Global Market Intelligence and not by S&P Global Ratings, which is a separately managed division of S&P Global.