OpenAI, the company behind ChatGPT, is taking significant steps to address the growing concerns related to catastrophic risks associated with artificial intelligence (AI).

As part of its mission to build safe Artificial General Intelligence (AGI), the organisation is placing a strong emphasis on risk preparedness. OpenAI’s latest initiatives include the formation of a dedicated preparedness team and the launch of the AI Preparedness Challenge.

Frontier AI models represent a leap beyond the capabilities of existing advanced models, offering the potential to benefit humanity significantly. However, they also bring with them increasingly severe risks. OpenAI acknowledges the need to address pressing questions related to these models:

1. How dangerous are frontier AI systems when misused, both in the present and the future?
2. How can we establish a robust framework for monitoring, evaluating, predicting, and protecting against the potentially harmful capabilities of frontier AI systems?
3. What would be the consequences if the weights of our frontier AI models were stolen and exploited by malicious actors?

To confront these challenges, OpenAI is committed to enhancing its understanding and infrastructure to ensure the safety of highly-capable AI systems.

OpenAI’s new Preparedness team, led by Aleksander Madry, is at the forefront of this initiative. This team is poised to tightly integrate capability assessment, evaluations, and internal red teaming for frontier models. Their work will encompass not only the models developed shortly but also those with AGI-level capabilities. The Preparedness team is tasked with tracking, evaluating, forecasting, and protecting against catastrophic risks that span various categories, including individualised persuasion, cybersecurity chemical, biological, radiological, and nuclear (CBRN) threats, and autonomous replication and adaptation (ARA)

A critical component of the Preparedness team’s mission is the development and maintenance of a Risk-Informed Development Policy (RDP). This policy will outline OpenAI’s approach to rigorous capability evaluations and monitoring for frontier models. It will also establish a spectrum of protective actions and a governance structure for accountability and oversight throughout the development process. The RDP is designed to complement and expand upon OpenAI’s existing risk mitigation efforts, contributing to the safety and alignment of new, highly capable systems, both before and after deployment.

OpenAI is also extending an invitation to exceptional talents from diverse technical backgrounds who are interested in working on Preparedness. The organisation is actively recruiting individuals to join the team and push the boundaries of frontier AI models.

Furthermore, OpenAI has launched the AI Preparedness Challenge, which aims to identify less obvious areas of concern related to catastrophic misuse prevention. This challenge offers a total of US$25,000 in API credits to up to 10 top submissions.