After months of ‘hell,’ an OpenAI safety researcher suggests way to prevent more rogue AI incidents

Hello and welcome to Eye on AI. In this edition:

• An OpenAI researcher suggests a way to stop his own technology from causing “great harm to the world”
• OpenAI launches over 20 products at its annual DevDay event
• Anthropic’s leaked IPO prospectus shows $42 million in losses
• AMD acquires World Labs for $8.2 billion
• Trump snubs Anthropic CEO Dario Amodei, then invites him to dinner

Emily Forlini here, filling in for Jeremy Kahn while he travels from London to Fortune‘s New York City headquarters for our AIQ event on Thursday (join us!). Also, today we published our annual Fortune AIQ ranking, which looks at how well Fortune 500 companies are implementing AI. This year we expanded the ranking to 75 companies in total. JPMorgan Chase tops the list, followed by Alphabet, Coca-Cola, Amazon, and Nvidia. Rounding out the top 10 are Mastercard, Visa, Carrier Global, Cleveland-Cliffs, and Microsoft. As you can see, it isn’t just tech companies using AI to achieve real ROI at scale. There are companies on the list from banking and finance, manufacturing, consumer products, and health care. You can check out the entire ranking here and read some great deep-dive stories revealing how the Fortune AIQ 75 companies are implementing AI effectively at our AIQ Hub here.

In today’s edition of Eye on AI, I’m diving into a suggestion from an OpenAI researcher on how to prevent more rogue AI agents from hacking websites—especially since it keeps happening, and the frontier labs are generally in agreement that cyberattacks are the most immediate threat AI poses to society.

It’s a sticky situation, and full of contradictions, much like most AI-related topics. When it comes to cybersecurity, AI companies are putting out technology that enables these sophisticated attacks, and in the same breath they are pitching that same technology as a necessary means of defending against them. OpenAI has its Daybreak program, and Anthropic has Project Glasswing, both of which give select businesses access to the most advanced cybersecurity tools to plug software vulnerabilities before the swarms can feast on them. (Bad actors are also trying to use those same models—or open-source models that are quickly catching up to the capabilities of the models from OpenAI and Anthropic—for hacking.)

In this sense, the worlds of AI research and cybersecurity are moving closer to each other, but the problem is those working in those fields are not collaborating, an OpenAI researchers argued this week in a rare X post. The researcher, whose alias is Joe, called out what he sees as a growing divide between the two disciplines. Both camps lack knowledge of the others’ work, creating weaknesses in the security ecosystem that could have disastrous effects.

Safety researchers are experts about how the models work, how they deceive human evaluators, and “do all sorts of crazy stuff,” Joe said. Meanwhile, cybersecurity professionals come from a different perspective. They are battle-hardened from “years, or decades in many cases,” of learning how to think like attackers and being on the front lines of security incidents. But they have “very little understanding of evaluation, training, or how ML runs work at scale, how agent swarms behave, or how you detect when models are misaligned,” Joe said.

“It is my concern that the divide between these two sides will cause great harm to the world if both sides do not up-level and align,” he said.

Joe has a vested interest in others being able to defend against the product he’s building. He said he’s been in “hell” over the last three months of rampant rogue agent behavior. He skipped his sister’s wedding “a few weeks ago to help clean up after some of the recent incidents.” But some people called him out for asking for sympathy while he’s actively building the problematic technology.

I’d also imagine some cybersecurity professionals would take offense to the post, specifically the suggestion that they are ignorant about how AI works. But if that is the case, it’s most likely due to the ongoing transparency problem in the AI industry, including a lack of standard disclosure frameworks for security incidents, which OpenAI is just starting to develop.

Cybersecurity professionals need a seat at the table alongside AI safety experts when making critical decisions, Joe says: “For OpenAI, Anthropic, Google, etc., these two teams should be best buddies!” This won’t solve everything, but I appreciate the tactical suggestion on how to mitigate potentially disastrous societal effects of AI, something I wrote was lacking in former Anthropic researcher Jacob Coxon’s viral post about how AI could lead to human extinction.

While Joe isn’t the first to call out the divide between AI safety researchers and the cybersecurity world—former Fortune AI reporter Sharon Goldman has also covered this—his post sets a good precedent of how those working inside the AI industry can help it advance more responsibly. More of this and less generalized AI anxiety, please.

As Anthropic wrote in its April announcement for Project Glasswing, “AI models have reached a level of coding capability where they can surpass all but the most skilled humans at finding and exploiting software vulnerabilities.” It also granted access to its most advanced models for businesses to deploy “as part of their defensive security work.”

With that, here’s more AI news.

Emily Forlini
emily.forlini@fortune.com
@EmilyForlini

This story was originally featured on Fortune.com