OpenAI shakeup highlights the impact of long-range AI risk

We had not planned to interrupt our report series on security for generative AI by zooming out so soon to the larger issues of AI risk — not least because any such discussion runs a high risk of becoming not just speculative, but of taking on a breathless detachment from reality. Events in recent days, however, have forced us to step back and consider more of the central and long-term issues of risk with the evolution of AI — the impact of these issues seems to be playing a part in highly visible events shaping the market right now.

What we mean by this is exemplified by the drama unfolding at OpenAI LLC and what we know already of its key players. Furthermore, what we can see, with respect to the impact of AI risk on these events, is less speculation than observation.

OpenAI has long been devoted to creating "safe and beneficial AI." As part of that mission, it has dedicated itself to mitigating the risks inherent in AI advancing from artificial general intelligence (AGI) to something even more comprehensive: "superintelligence" (in the company's words), which potentially poses risks beyond the ability of humanity to manage them. While today's risks of generative AI manipulation and the exposure of sensitive or objectionable output may suggest AI's current exposures, it is the evolution of AI itself that is the more daunting challenge. While the true dynamics underlying the shakeup at OpenAI may not be known in detail anytime soon (if ever), there is little doubt that the tension already evident at OpenAI about how best to execute its safety mission exemplifies the issues at the heart of the AI risk challenge — issues that will radiate outward not only to OpenAI's immediate partners, such as Microsoft Corp., but to the rest of the industry and beyond, to society as a whole.

OpenAI turnover seen through a risk glass (darkly)

Indicators of that tension present in the dismissal of Sam Altman include the remit of OpenAI co-founder and chief scientist Ilya Sutskever. Sutskever is also a member of OpenAI's board, and his role in Altman's firing has been much discussed. A few weeks ago, Sutskever obliquely tweeted that "ego is the enemy of growth." Altman's dismissal by the board places that comment in a new light.

One of OpenAI's most prominent efforts under Sutskever introduced this past summer is the company's "superalignment" initiative. As defined by OpenAI, superalignment refers to the solution necessary to maintain control over superintelligent AI. This is an evolution of AI beyond the next major horizon of AGI. If AGI can be compared to human intelligence in meaningful ways, superintelligence refers to capacity beyond that of humanity.

For (hopefully) understandable reasons, it is daunting to write about this as a credible analyst of the industry as it is today. However, the potential of superintelligence and Sutskever's role in leading OpenAI's efforts to direct and control it may well be having an impact on the leadership dynamics of an organization such as OpenAI, given its distinctive combination of nonprofit parent and for-profit arm, as well as the altruistic aspects of its mission. If OpenAI's leadership is divided on the priority of its altruistic goals relative to others, it may be a serious enough issue to have a hand in forcing a messy divorce such as this.

Other factors are centered on AI's abundantly evident commercial potential, as well as OpenAI's unique role in that aspect of AI evolution. One of OpenAI's most visible commercial partners, Microsoft, recently held its Ignite event, during which it released a host of new offerings that both capitalize on generative AI and seek to contain its risks. Ironically, Ignite was bracketed at the beginning by OpenAI's Developer Day and at the end by the firing of Altman.

OpenAI has reportedly been valued most recently at as much as $80 billion or more in anticipation of a further tender of shares — nearly triple its valuation less than six months ago. The dilemma for anyone pursuing OpenAI's value for vested self-interest is its potential conflict with its avowed ideals for benefiting society as a whole. Few have placed as much at stake in the commercial value of OpenAI as Microsoft, which has pledged another $10 billion in addition to the $3 billion it has already invested in the company. Among the salient aspects of that $10 billion deal: 75% of OpenAI profits go to Microsoft until the funding is recompensed, in addition to what OpenAI pays for the vast Azure compute resources it requires.

Appearing onstage with Altman at OpenAI's Developer Day, Microsoft CEO Satya Nadella affirmed the value of the two companies' partnership. After the recent shakeup, Microsoft reiterated its confidence in OpenAI, and then made moves to hire Altman, Greg Brockman and other OpenAI staff in the days following Altman's firing. (Altman has since agreed to return to OpenAI.) Beyond the immediate crisis, however, it would seem difficult to underestimate the perceived impact on the relationship between Microsoft and OpenAI, and Microsoft's plans.

What does superalignment mean?

So what is at the heart of what OpenAI says it needs to out-innovate intelligence potentially beyond human and keep it from (as OpenAI describes) "going rogue?" As a starting point, OpenAI points to current techniques available to align AI with acceptable intent, such as reinforcement learning from human feedback. When superintelligence grows beyond the ability of humans to maintain this alignment, efforts must rely on a level of capability effectively equivalent to the superintelligence it seeks to oversee, hence superalignment.

Three objectives identified by OpenAI in its introduction of superalignment are the development of a scalable approach to training, validation of any model that results, and testing its capacity and efficacy across the entire pipeline of techniques applied to AI alignment.

When the evaluation of AI systems exceeds human capacity, it is AI that must assist in the assessment of other AI. This concept of "scalable oversight" is implicit in patterns we have already identified, such as the "quarantined" large language models deployed alongside, or in line with, the "privileged" LLMs they are intended to protect. The aim is to generalize these learnings to tasks humans can't supervise. Robustness is also a goal, meaning the ability to automate the identification and interpretation of concerns.

A final superalignment objective is evident in events observed over the past year: the subjection of a system to adversarial testing. While adversarial approaches are well known in cybersecurity for techniques such as "red team" penetration testing of system confidentiality, integrity and availability, stress testing is well known in fields such as IT performance, and the simulation of unusual incidents and emergencies in the physical world. The suggestion of adversarial testing, however, brings to mind the efforts already seen in the past year to subject generative AI to such evaluation — among model providers as well as at scale, as in the example of the DEF CON Generative Red Team Challenge this past August.

The investment contemplated by OpenAI in this effort is not insignificant: 20% of the compute the company has secured over the next four years. Not a small piece of priority for a company that had already sustained a $540 million loss during 2022 while it was developing ChatGPT, and which may need to raise as much as $100 billion in the coming years to achieve AGI capable of improving itself, according to reports of private comments made by Altman earlier this year. Compounding these dynamics is the fact that superalignment is now Sutskever's core research focus, according to OpenAI's July announcement of the initiative.

A driver at the deepest level

It may be a cliché to say that the importance of a goal such as superalignment cannot be overstated. In the froth of AI noise, things are overstated all the time, but the fact remains that considerations of the potential impact of superintelligent AI cannot be dismissed, and time is not our friend in taking whatever steps we can to manage that impact.

For those concerned about AI risks, this puts the recent events at OpenAI in perspective. No matter what the proximate issues may have been that led to the high-profile turnover in OpenAI's leadership, its key stakeholders may well differ on priorities that, seen in this light, may not just be critical — they may be seen as existential. To those wrestling with them directly, it is worth whatever pain may be seen as necessary to not place priorities so vital to the future of humanity at risk.

This introduces the further risk of dependence of so much of technology's future on the small number of innovators able to mount the expertise and resources necessary to advance AI at this level — and in Microsoft's case, at least for now, primarily one: OpenAI. It is an ironic turn of events for the company that, over the last 30 years, has become a primary provider of technology to businesses around the world.

^{This article was published by S&P Global Market Intelligence and not by S&P Global Ratings, which is a separately managed division of S&P Global.
451 Research is a technology research group within S&P Global Market Intelligence. For more about 451 Research, please contact.}