The leading AI companies are increasingly focused on building generalist AI agents — systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. In his upcoming talk, Yoshua Bengio will discuss how these risks arise from current AI training methods.
This lecture is part of the Large Language Models and Transformers Thematic Semester that is Co-Organized and Co-Funded by the Simons Institute for the Theory of Computing and IVADO taking place in Berkeley from February to May 2025.
The Lecture will be live-streamed and will take place on Tuesday, April 15, 2025 at 7pm ET (4pm PT).
This talk will be followed by a panel discussion with Professors Yoshua Bengio (IVADO – Mila – Université de Montréal), Dawn Song (UC Berkeley), Roger Grosse, Geoffrey Irving, and moderated by Siva Reddy (IVADO – Mila – McGill University).
Registration is required to access the livestream, or for early access to the recording.
Abstract
Various scenarios and experiments have demonstrated the possibility of AI agents engaging in deception or pursuing goals that were not specified by human operators and that conflict with human interests, such as self-preservation. Following the precautionary principle, Bengio and his colleagues see a strong need for safer, yet still useful, alternatives to the current agency-driven trajectory. Accordingly, they propose as a core building block for further advances in the development of a non-agentic AI system that is trustworthy and safe by design, which they call Scientist AI. This system is designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans. It comprises a world model that generates theories to explain data and a question-answering inference machine. Both components operate with an explicit notion of uncertainty to mitigate the risks of overconfident predictions.
In light of these considerations, a Scientist AI could be used to assist human researchers in accelerating scientific progress, including in AI safety. In particular, this system could be employed as a guardrail against AI agents that might be created despite the risks involved. Ultimately, focusing on non-agentic AI may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory. Bengio and his colleagues hope these arguments will motivate researchers, developers, and policymakers to favor this safer path.
Yoshua Bengio is a full professor in the Department of Computer Science and Operations Research at Université de Montréal, as well as the Special Advisor and Founding Scientific Director of IVADO and Scientific Advisor and Founder of Mila . He holds a Canada CIFAR AI chair and is the recipient of the 2018 A.M. Turing Award, considered the “Nobel Prize of computing.”
He is a fellow of both the U.K.’s Royal Society and the Royal Society of Canada, an officer of the Order of Canada, a knight of the Legion of Honor of France, and a member of the U.N.’s Scientific Advisory Board for Independent Advice on Breakthroughs in Science and Technology.
The Richard M. Karp Distinguished Lectures were created in Fall 2019 to celebrate the role of Simons Institute Founding Director Dick Karp in establishing the field of theoretical computer science, formulating its central problems, and contributing stunning results in the areas of computational complexity and algorithms. Formerly known as the Simons Institute Open Lectures, the series features visionary leaders in the field of theoretical computer science and is geared toward a broad scientific audience.
The lecture recording URL will be emailed to registered participants. This URL can be used for immediate access to the livestream and recorded lecture. Lecture recordings will be publicly available on SimonsTV about 12 to 15 days following each presentation unless otherwise noted.
