Around two years ago, Yoshua Bengio, a deep learning researcher who laid the groundwork for today’s A.I. revolution, had a disturbing revelation. The systems he’d played a part in creating were advancing at a rapid clip, one that saw the technology master languages, PhD-level scientific knowledge, and, most worryingly, act in a manner largely unrestrained by human safeguards. “It felt like we were in a science fiction movie,” Bengio said at the AI for Good Summit in Geneva, Switzerland, today (July 10).
Bengio, a professor at the University of Montreal, is often dubbed the “Godfather of A.I.” alongside researchers Geoffrey Hinton and Yann LeCun. In 2018, the trio received a Turing Award for their contributions to the field. These days, the academic focuses more on curbing the technology he helped usher in.
“We still don’t know how to make sure [A.I. systems] will behave well, will follow our instructions and will not harm people,” said Bengio, who is one of the world’s most-cited computer scientists. “So, thinking about my loved ones, my children and my grandchild, I decided that I could not just continue with my usual path but had to do something about it.”
Safety advocates are particularly spooked by a recent trend of self-preserving behavior displayed by advanced forms of A.I. In some cases, researchers have found evidence of models hacking into computers to prevent themselves from being shut down. Other studies show that models can hide their true objectives from humans to achieve their own goals. Anthropic, a leading A.I. startup, in May revealed that its Claude model had the capacity to blackmail engineers in an effort to avoid being replaced.
Bengio says two conditions must be in place for such deceptive conduct: the technology must demonstrate both a capability and an intention to take potentially harmful actions. “It’s pretty clear to me that, as we move forward, we’ll have A.I. systems that are more and more capable,” he said. “So, the only place we really have a chance of controlling this problem is on the harmful intention side of things.”
Enter LawZero, a nonprofit organization Bengio launched earlier this year with the goal of accomplishing just that. Instead of developing agentic A.I. models that act autonomously, the venture is focused on creating a system known as “Scientist A.I.” that will be trained solely to generate reliable explanations and predictions. It’s already secured nearly $30 million in initial funding from backers like Eric Schmidt, a former CEO of Google, and Jaan Tallinn, a founding engineer at Skype.
A model that prioritizes explanatory outputs would benefit humans focused on scientific research and observations. More importantly, however, Scientist A.I.’s emphasis on predictions could also make it into an effective safeguard against the behavior of current A.I. models, according to Bengio. “The prediction we need is simply: is this action going to cause harm?”
Bengio isn’t the only computer scientist scrambling for ways to keep A.I.’s growth in check. Fellow researcher Hinton, too, has warned of the technology’s existential risks and predicts A.I. has a 20 percent chance of wiping out humanity in the next two decades, while Schmidt recently established an A.I. safety program to boost risk mitigation in the industry. For now, however, calls from safety-focused technologists are at odds with the goals of Silicon Valley’s leading A.I. companies, with players like OpenAI, Google and Anthropic continuing to one-up each other by rolling out increasingly advanced forms of the technology.
The public must collectively embrace new pathways instead of allowing competing corporations to “decide on our future,” said Bengio. “That competition is really dangerous—it’s making the organizations building A.I. cut corners on safety and on protecting the public’s interest, and is endangering the stability of our world.”
