Negotiating AI Safety Pacts with China

Kara Kara, I was trying to do this sort of optimistic, you know, ending riff and you're like Sebastian. You just burst my balloon. You know, intelligence has its limitations, but stupidity and greed are infinite. So that's my feeling. I'm sorry. Hi everyone from New York Magazine and the Vox Media Podcast Network. This is On with Kara Swisher, and I'm Kara Swisher. My guest today is journalist and author Sebastian Malaby. He's a longtime chronicler of power and innovation, especially in the world of finance and economics. For his latest book, The Infinity Machine, he turns his attention to tech. His central figure is Demis Hisabas, the CEO and co-founder of Google's AI R D lab, Deep Mind, and a Nobel Prize winner in chemistry . Hasabas has dedicated his life to using AI to unlock the mysteries of physics and biology. A former child chess prodigy, he's rapidly competitive and insatiably curious. But Malaby says Demis is also one of the very few big names in AI development who genuinely cares about public safety. His struggle to balance his ambition, his personal goals, and the realities of the corporate AI race is, according to Mallaby, one of the most defining stories of the era. I think it's really important to look at characters like Damas because he was a very early person in AI, at least the modern version of AI, because it's been around forever. also based in London and kept his company there and away from Silicon Valley. So he has different goals here. At the same time he's incredibly ambitious and a bit crusty, according to lots of people, very typical of a science researcher type, but he has less and less power over what's happening because of the vast amounts of money pouring into the space. People tend to forego safety for profits if in fact they make profits anytime soon. He is different from other AI developers, though there's a lot of people who are sort of in his area, but he's certainly one of the leading minds of this age and someone you should know well. All right, let's get to my interview with Sebastian Mallaby. We have two expert questions today from an ASDAC CE O, Adina Friedman, and Kent Walker, President of Global Affairs at Google and Alphabet. This interview was recorded live in front of a virtual audience. series at American University's COGOD School of Business, so stick around Support for this show comes from The Guardian and their new show, Stateside, where journalists Kai Wright and Carter Sherman use the entire independent reporting resources of The Guardian to slow down the news and wrestle with the questions we all have about what is actually happening in the world. Join Kai and Carter three times a week as they utilize all the reporting resources the Guardian has across news, international coverage, climate, culture, wellness, and more. And the Guardians, not owned by a billionaire, they fearlessly report the facts without interference. Go to theguardian.com slash stateside to learn more and listen wherever you get your podcast or watch on YouTube. That's thegardian.com slash stateside . What's up y'all? I'm Skylar Diggins, seven-time WNBA All-Star, Olympic gold medalist, and mom. And I'm Cassidy Hubbard, host and reporter for nearly 20 years, covering the biggest names and stories in sports and mom . And this is and mom, a community for athletes, game changers, and moms of all kinds. Dropping May 14th. Tap in with us . Sebastian Mallaby, thanks for coming on on. Great to be with you, Kara. So for people who don't know who Demis is, it isn't a household name the way other major AO figures have become like OpenAI's Sam Altman, Anthropic's Dario Amoti, or of course XAI's Elon Musk. Even though he's accomplished as much or more in AI, I'll just say I broke the story when they bought his company uh at Google w a long time ago. And I met him just briefly during that he was mad at me because I broke the story. But nonetheless, let's put him into context, cause I did understand his the importance of him in AI and he definitely he's not in the shadows, but he's certainly not as well known and he doesn't put himself out there. So how is he like them and what sets him apart So the first thing that marks him out is that he was the first, right? So the first scientist entrepreneur sets up deep mind in 2010 when AI can't even recognize a cat photo. Nothing worked, deep AI winter. And then the others came afterwards, and they came afterwards as straight derivatives, right? So, you know, open AI is set up five years later, and Elon and Sam are explicitly trying to do the anti-demis , anti-Google Deep Mind company. Oh yeah. And then you go forward uh anthropic and you know, Dario as a PhD scientist, wanting to sort of you know do it with more of a social conscious and make it safer or like says I think also admired Demis and his you know people at anthropic say sometimes he's the one who is the closest to a model that Dario had in his head. So first point is he's the original demis. He's the only one with a Nobel Prize. He's different in that because he started early, his approach to AI was not merely to scale an existing technology path, but actually to invent the technology path. And so they brought together agentic systems from reinforcement learning, mixed that up with deep learning, learning from data, and kind of invented the field. And I think there's some DNA left over today from that experience where if it was to be the case that AI in the next three years went down a novel path, something more than just scaling the transformer architecture. It would be more likely to come from Google DeepMind and Demistosabis than from the rivals. So talk a little bit. Also, by the way, AI has been around for a very long time. He started in En gland, that was one of the things. Demas started the company there. And of course Alan Turing. Hello. Nice to meet you. But a lot of stuff went on very early in England actually, which is interesting. But how is he like the others for better or worse? How do you what if they look at him as a immortal or someone that they look up to or are trying to copy in some ways? Talk about the attributes and maybe some of the negatives about that. He is different I think in that his motivation is this intensely, intensely scientific curiosity, to the point where he expresses it to me in spiritual language. And there was this moment, you know, fairly early on in my more than thirty hours of conversations with him, where we were sitting in a park in London and at the next table there were people having regular chats about their friend who went to hospital or whatever it was. And I was opposite this sort of messianic possessed person who was banging the table and saying, you know, when I'm reading a scientific paper and it's two o'clock in the morning, reality is screaming at me, demanding to be discovered, demanding to be understood. And and if I can understand it and get close to the fabric of reality and and nature , then I will understand the intelligent being that that might have created this and I'll be closer to God. And so or at least his understanding of God. So that is a uh something which is very distinct, right? Some of them are getting spiritual. Peter Teal talks about the Antichrist, which is anyone who's against AI. You know, e i when you said that when he's doing that, that actually would trouble me because these people do tend to think of themselves as gods manipulating machines or in some fashion. And so a lot of it is indeed spiritual, and at the same time , they're not gods, they're people, correct? Correct. And I mean this gets to this question of, you know, is the motivation simply the sweetness? This is the quotation from Robert Oppenheimer, the Manhattan Project. The quote was, you know, repeated and at the time Jeff Inton didn't know that he was being overheard by a journalist from the New Yorker, but he was. And what he said was, you know, he repeated Robinheimer's line, I'm not creating AI for any particular good reason. I'm doing it because when you see a technology that's sweet, you just do it and you figure out the consequences afterwards. And as a scientist I can't resist. And I think Demis has quite a lot of that. Um and so except Oppenheimer was a little more thoughtful, wouldn't you say? I mean, I have become death or that kind of thing. Uh I don't know. I mean um Demis uses some pretty funky philosophic references too. He talks about Spinoza, he talks about Immanuel Kant. He can riff on anything. Yes, comparatively he's quite like deep. So is Dario Emo in many ways. I mean Dario Modi can quote from plays, which none of these others can do. Um Elon's mostly sci-fi crap and Sam not nothing at all, although he's not stupid by any means. But Demis comes from a working class immigrant family. As a kid he was a chess prodigy, not a surprise, he was consumed by the idea of AI. Around seventeen he said he decided to dedicate his life to this. Talk about what drove that singular focus to be the persons who solve artificial general intelligence and explain what that means because that's defined differently by different people. I want to hear how you see his definition of it. And this is called AGI. And again, you write that its definition remains fuzzy. So talk about his definition. Sure. So just just on the kind of early motivation, it really was extraordinary. I mean to at seventeen and we're talking kind of 1993. You know, AI was talked about since Alan Turing in the 1940s, but it went through these rises and falls, as you know. And this was definitely winter at that time. And you know, the kind of AI that was happening was academic s fiddling around with toy experiments. It had no relevance to the real world at that time. So to believe not only that you were going to devote your career to AI, but to super powerful AI, artificial general intelligence, which, as you say, is defined by different people, but we could we could say it's the ability for an AI system to be smarter than humans at any screen-based task. I think that's probably a sort of average type definition . It's not the most extreme or not the most modest. So, you know, this is what he wanted to do when he was 17, which is pretty amazing. Aaron Powell Right. And that's how he defines it. Any screen-based task of figuring out. Well, actually, you know, the f the f funny thing about Demis is that he so loves the process of discovery and of research and the quest. You know, he's got one Nobel Prize, but he definitely would like another one. That he loves to go for a definition of AGI these days , which puts it out further into the future so that the joy of the journey can be extended. And so what he says is if you could train a computer or an AI system on everything that was known in nin eteen eleven and then you waited to see if by itself it would discover general relativity and then it did, that that would be AGI. That's this new definition. Ah, probably Maximalist. Probably it couldn't you? Yeah, I guess. Wow, that would be something. Uh sorry, I doubt that would happen, actually. I'm going with Einstein on this one. Um he co-founded Deep Mind in 2010. Obviously the name Deep Mind is exactly what he's going for. Four years later, as I said, Google bought it for six hundred and fifty million dollars. He was able to extract big concessions in the sale. I remember writing about it. It would remain in London, the offices, and there would be restrictions around the use of their technology. Talk about being in London because most people working on this stuff were in whether it was Fei Fee or Hinton. He was at Google, Fei Fe Lee was at Google for a little bit and then elsewhere. Some of the early people were deep in Silicon Valley. So why stay in London during this sort of social media smartphone boom at the time? When I wrote about it, everyone was like, What's that? And I'm like, no, no, this is a big friggin' deal for Google to buy. I th I understood what he was doing there, but talk a little bit about why he sold it. Obviously need the money to grow it. I think because as as it's turned out, this needs a lot of money. W talk about remaining in London and why. Yeah. So you know, Demis' official explanation for why he stayed in London was that there was a lot of scientific talent in London and in Europe, and you know, if you were trying to recruit great scientists, you had a competitive advantage of hoovering up that, you know, geography. And you know, if you were in Silicon Valley, it'd be super competitive to get the best people, and it would be, you know, better to be in London. When you actually look at who he recruited , the first PhDs who came, they were sort of from all over the place. They came from Canada, from people who'd studied under Jeff Hinton in Toronto. They came from Switzerland, where there was a good PhD program in AI . They you know there was Corey Kavak Choglu who studied in New York and uh Jan Nikon . So this notion that it was for people who don't know, Jeff Hinton is considered one of the early godfathers of modern AI, also Faye Fei Li is. And and Jan Lakun later ran until recently Meta's AI effort. Right. And he was also an academic kind of pioneer. And and the point is that that what Demis claim ed as being the reason for staying in London was not what how it turned out. Because he was recruiting from all over the world, including from the US. And so I actually think the reason why he stayed in Britain is simple patriotism. He just likes Britain. You know, he's a classic melting pot product of immigrant parents in London, very attached to London, and he simply thinks that the values in Silicon Valley su Yeah, I n I I've heard him say that. So the mentality around the company was very different because it was happening and especially at that moment when things sort of started to go off the rails, I would say 2010, 2011 is when Silicon Valley started to lose its ever love and mind. So, you know, beyond the move fast and break things nonsense, it really did. It became something much different. And that didn't exist in London, obviously, and the safety demands were the biggest concession for Google. Deepmind demanded a ban on military uses, think about Dario Modi at this point. Demus and his team also insisted on forming an outside safety review board to dilute Google's influence over the technology , smart. Why did Google ultimately agree to those terms given it was obvious how important AI was to the company's future, especially to their core business search, which this would decimate essentially in its current yeah yeah I mean the the point person in the negotiation with Demis r in that transaction was actually Larry Page. And and that was important because Larry, you know his father had worked academically on deep learning. And that impressed Demis. And Demis felt that if he sold the company to Google specifically, he would be in the hands of somebody who loves science. Larry, you know, he once said to me, you know, you could imagine Larry as a professor at a top college like Stanford. And from Demis , that's the ultimate comment of respect. So whereas, you know, Mark Zuckerberg tried to buy Demis and Deep Mind, and Demis sort of laughed him off, and Elon tried to buy DeepMind and Demis refused. He was happy and comfortable with Google because of that scientific culture. As they would be, because these were two PhD candidates themselves and Google And then came the conditions, as you say, around um safety uh around the AI, uh sort of safety oversight board. And I did speak to the the sort of chief lawyer who was on the MA team at Google, who repr remembered sort of basically being you know terrified by the idea of you know diluting Google's right to Correct. Today it's nothing. It's like some credit card card But this this lawyer said to me, you know, you we have a fiduciary duty as a public corporation to own assets from which we derive value. And if we're told, well, you can't derive value 'cause you have to have this outside bunch of safety overseers who are gonna tell us when you can or cannot deploy. Forget it. I mean that's are we even allowed to sign that kind of deal? And the way he described it is that the reason that Google caved and gave DeepMind what it wanted , is that whereas with many startups that Google buys, the notion is they've got some good tech and we'll get rid of the fan if we don't like him. To the contrary with DeepMind is they wanted Demis. Like he personally was the reason they were so keen on paying up. And by extension, they were willing to give him what he wanted on safety . We'll be back in a minute , I'm support for this show comes from Bowl and Branch. If there's even one thing off about your bed, it'll show up in your sleep. Maybe your sheets slip around too much or your pillows not quite right. These issues only get worse over time as materials wear down, which can wear you down alongside of them. With Bowl and Branch, you can upgrade your bed and find everything you need to get better sleep. They make everything from cotton sheets to pillows to blankets and comforters. All are designed to be breathable, incredibly soft , and get better over time. Most people start with signature sheets, and it's not uncommon to buy two sets of sheets so you can rotate them with bowl and branch. The moment you get into bed, you notice the difference. I have tried out the bedding. It is fantastic. I have a waffle blanket that I love so much, especially in the summer, where it's cool and warm at the same time. So get 20% off your first order plus free shipping during the memorial day sale at bol andbranch.com slash Kara with code Kara , that's Bowl and Branch, B O L L A N D Branch, B R A N C H dot com slash Kara, K-A-R-A. Code Kara to get twenty percent off bowl and branch dot com slash Kara code Kara exclusions apply . When you travel well, your KLM Royal Dutch Airlines ticket takes you to more than just your destination. It takes you to front row views, voices lost in the music, and new shared memories. And when the last song fades, welcome aboard KLM Royal Dutch Air. KLM Royal Dutch Airlines crew is here to ensure your journey home hits all the right notes. The fast and seatbelt side. KLM Royal Dutch Airlines. When you travel, travel well. Support for this show comes from The Guardi Trying to locate news sources that reliably separate fact from fiction can seem like looking for a needle in a haystack these days. Social media feeds, cable news, and subscription-based news outlets might be able to give you one version of the news, but not in the way that you need. You want news fast, but slow enough to actually absorb the information. States ide with Kai Wright and Carter Sherman is here to do just that. The podcast tries to slow down the news and wrestle with the questions regarding the world at large at your own pace. Three times a week, hosts Kai Wright and Carter Sherman bring their expertise together to showcase differing and engaging perspectives regarding the world today. They will use their collective knowledge to discuss the news, international affairs, climate, culture, sports, lifestyle, fashion, and wellness. Plus The Guardian wants to emphasize they're not billionaire-owned, meaning they're free to report the whole picture without interference. Go to theguardian.com slash state site to learn more and listen wherever you get your podcast or watch on YouTube starting May 13th. Now he had hoped there'd be one collective effort to v develop AI, because there wasn't a lot of AI development. As you said, it was a w winter of AI at the time. And he had hoped there'd be sort of a Manhattan project with Google and DeepMind at the center. And you know, one one of of the the early interviews I did with Sam and Musk when they were starting OpenAI was their worry that deep mind in Google would dominate. Not just them, but possibly Mark, who they had thought little of at the time in that regard. And so he he was shattered after the first ethics and safety group meeting in 2015. Musk hosted it and then teamed up with Sam to start open AI when they were getting along. And it was a rival. And it was started. He is he was testifying honestly for a second when he talked about the original reasons. I happened to write that story when that happened when Open Mind was created So talk a little bit about the fallout of the meeting and how it changed his approach to AI development and the way he ran DeepMind. Yeah, I mean so what happened as you say is that there was this uh safety oversight meeting which was supposed to be for Deep Mind to get useful feedback, and it turned out the people at the meeting were not giving useful feedback, they were kind of learning how DeepMind was doing stuff and then collecting information. So so in that meeting there, was both El on but also Reid Hoffman, who was one of the funders of OpenAI. And so the reaction was from Google: forget the idea of safety oversight boards. We've tried that once and it was terrible. So we're not going to do that again. And Demis's reaction was: well, we can't give up on the idea of a safety oversight board. It's essential for this technology. And so we're gonna have a fight about it. And so now this is you know 2016 is when this really gets going, and there's this thing called Project Mario, as they called it internally at DeepMind, where for three years they retained, you know, armies of lawyers and investment bank strategists and people like that. And there were term sheets running to 50 pages that flew back and forth between the uh you know negotiators and mountainw Vie and the negotiators in London. And I was shown some of these things. He tried to spin it out. He tried to spin it out at the end. Yeah I mean he thought of spinning out because that would be the threat that would force Google to agree to the safety oversight board. And, you know, Reed Hoffman promised a billion dollars to finance a spin-out. They went to see Joe Tsai, uh the co-founder of Alibaba in Hong Kong to try and get some money out of him. It was a real thing . And you know, what happened after three years is that Dem is caved. He was so exhausted by this three years of negotiation and, you know, he'd get another long term sheet from the lawyers and he would hold his head in the hands and and say I don't want to let him do it. I don't want this part of my brain to expand is what Demis wound up saying about legal documents. And so the upshot was, you know, he didn't get what he wanted. Google was never gonna do it. I ca I kind of developed a weird respect for Sunda in learning about this process because he was very canny. You know, he knew he didn't want demis to spin out. He didn't say it quite directly , but he, you know, hid behind his chief counsel and other people who played bad cop on his behalf. And at the end of the day he got what he wanted, which was to keep Demis inside the tent, not spun out. Because he under he did understand what was happening in Silicon Valley with the competitors, right? Google had to be at the forefront of this. And uh Sundar, who's who's a very lovely affect, is really good at playing rope a do But let's talk about this idea of what Demas wanted to do here. He says he was naive to believe in a singleton scenario where AI developed collectively, but he also saw himself at the center. of the effort Now that you could say that's egotistical. And it does take a certain amount of narcissism that you should be in charge of all AI development. I have problems with I am more with the collective, but at the same time , it always di devolves into either one person or just a few people developing this thing. But talk about this idea because they they were never going to do it once they understood the financials here, the amount of money and the amount of power they would was that naive of him or ego maniacal? He says he was naive. Yeah, I I think it was naive of Demis to believe that AI would be developed by one single lab on behalf of all humanity. I mean humans are just disputatious and tribal and jealous and they don't do that. Greedy mostly. But you know I I like that you said probably there. No I said probably because because what what strikes me, Carrie, about this whole field is that the normal cynical explanation that people are doing it for the money is weirdly less true because AI is such a weird technology. It attracts people who are who are basically doing it for power, for ego, for power is yes. It's Promethean. It's not about money. So, you know, in hindsight, why did he not see this at the time? And and I don't think he was doing it for the money. He is a true believer of all these people. And of course, there's the glory. He wants a Nobel Prize. That's pretty great. And he has curiosity. But should he have been less naive or explain the naivete because it seems obvious to see that they were not gonna do this. Yeah, I mean I think there's you know there's some you know, mitigating circumstances that you can cite that make it a bit less crazy to have believed in this single thing. And so there was a sense of a single community at that point. And then you, know he, starts Deep Mind and it's going really, really well, and he does, you know, this Atari model, which is just an ex an astonishing agent that can learn by itself how to pay, you know, dozens of different video games and that impresses everybody and then he gets you know the Google checkbook behind him and it feels like he's the only game in town. Um and if there's a second game in town it, would actually be Google Brain in Mountain View. And so that is the same company. And you know, OpenAI just wasn't a thing then and didn't really become a thing, you know, until let's say twenty nine years. Yeah.. Yep Um so so I think it wasn't totally crazy at the time to think, well, I can't see anybody else on the horizon who's doing this, so it's gonna be me. But it's still crazy, still naive, still you, know, immature to think that when you're confronted with a godlike technology, there won't be lots of acolytes trying to do it and there'll be sectarian splits and you can continue that metaphor. Right. Now we've heard Altman Musk and even Mark Zuckerberg claim they care about AI safety. I d believe none of them, only to watch them prioritize growth and their own egos of time after time. Is is he different? Is he remain different? There's almost no incentive for tech companies like Google to make AI safety their top priority. For example, Google is eager to supply AI to the Pentagon after agreeing not to do so when it bought DeepMine. So you meant while he understands, Demas understands the risk of developing AGI. Does he have any more power to contain them in the face of the pressure? Both national security, financial, just to win, to be the dominant technology of the next era, et cetera. Yeah, I think Demis has gone a long journey, right, since founding DeepMind 16 years ago. You know, at the beginning he thought there might be a singleton scenario. Then when that started to break down because OpenAI, you know, launched a competitor , he still hoped to get oversight for his technology as we discussed with Project Mario negotiating. And then when that didn't work, he said, Well at least I'll make AI good for humanity by doing it for science And then, you know, along the way, although interestingly he didn't ever tell me this, I only know it from other sources, he was the one who said to Rishi Sunak, the British Prime Minister, you know, let's have a global safety discussion at Bletchley Park in 2023, and that happened. Um and so I think he had a reasonable trying to explain the why Bletchley Park. That's where Well that was where during the Second World War the German code was broken with by tearing with sort of the early kind of precursors of AI. Exactly. So so I think, you know, up to 2023 and that summit, he had a pretty good track record, Demis, in you know, walking the walk as well as talking the talk on safety. I think though that once he got into the race full on to do chatbots in competition with you know OpenAI and then Anthropic. He's he's done less and less, at least visibly. And it strikes me, for example, that by his own logic, open source models are dangerous, and yet Google releases them. He wanted to deprive the military of AI, and yet Google now supplies them. And his sort of rationalization for this is simply that there's a race on Chinese labs are part of the race. If he were to step back, if he were to quit, it would make zero difference to outcomes for humanity because there'd be six other labs doing Right. I mean he I think his power had waned is in that regard because people caught up, even if they're not as brilliant as he is. Now, he reminds me a little of Nikola Tesla, ultimately, you know, the greatest mind in Edison just ran right over him. Um, in 2024, he shared the Nobel Prize in Cameroon for his work using AI to predict structures of proteins, as you noted. Amazing thing. It could lead, it will lead to the creation of new drugs and vaccines to fight diseases. An amazing thing. And an astonishing use of AI in in service of humanity, I think one of the best examples, and not marketing that the rest of them foist upon me on a daily basis. And it could be where where AI makes people's lives better. Talk about this because and what it can mean for medicine. I know a lot about this. I've done a lot of reporting on this, but it really was a moment and it is taken off, whether it's mRNA, all kinds of stuff, drug discovery, you know, the quickening of drug discovery and ideas . Talk a little bit about this. Yeah. Well I know your CNN show gets into this, right? And so you probably could talk about it more than I can. But I mean No, no. Certainly in the creation of AlphaFold, this system, I mean it that in itself is an amazing story where Demis at Cambridge as an undergraduate had been told by a biologist friend, oh there's this conjecture in structural biology by a Nob laureelate from the nineteen seventies called Christian Amphinson, that if you stretched out an amino acid chain and you looked at the DNA sequence on the chain, the sequence tells you how that chain will fold itself up like a self-executing orig ami model into an intricate, beautiful shape, which is a protein. And we have proteins in our bodies, and there are proteins in plants that they're basic the building blocks of nature. And you can tell this intricate structure just by looking at the code. And so it became a sort of grand challenge in biology. Who can create the computational system that will just look at the code and then guess the shape. And from the 1990s, there were teams in different universities competing to do this. And DeepMind decides: after winning Go, defeating the Go champion in twenty sixteen. Well that was a game. The game of Go. Yeah, exactly. That was just a game. But then having solved that problem which involved this massive combinatorial complexity, because Go has all these different permutations. They do, but boring in comparison. Boring in comparison. So why don't we move on? Checkers. And and and leap all the way to predicting the combinatorial complexity of these possible shapes. You know, one strand of amino acid could fold itself up in billions upon billions of different shapes. So it was in that sense it was similar to Go, and having solved Go, they felt that their computer science skills had reached a point where maybe protein folding was crackable. And so they set out to do this in 2016. By 2018, they'd beaten all the universities, they had the best model . But it wasn't good enough that a you know pharmaceutical research team could simply use the prediction from computation to create a drug. So then the question was, should we push on and try to really make it that accurate? And the view of the leader of the team within Deep Mind was, forget it, boss. We can't do this. This is impossible. Don't just send us down a blind alley. We should just declare victory. We're the best team in the world. And then we should move on. And Demis was like, No, uh, I want you to actually solve the problem, not just be the best. And so they had a bit of a fight about this. And Demis said to me, Well, I was being unreasonable, but I wanted to be reasonable in my unreasonableness. So I listened into the team's discussions for a wh ile. And when I heard they had lots of ideas that were just flowing naturally, which they hadn't yet tested in the lab or in the computational lab, I figured, well, there's more stuff they could try. And so then he pushed them, he switched out the team leader, put a new person in charge, and then in twenty twenty they they succeeded and and they did predict the shapes of proteins. And then the the kind of the badass move was to open them source it, just like, you know, for free, any scientist in the world can effectively do a Google search and get the shape of the protein that they want to work on . We'll be back in a minute. Pregnant athletes are not fragile. Yeah, that's right. I said pregnant athletes. I am Rabinaj San, VP and head instructor at Peloton, and I PR'd my deadlift the week before my son was born. I was also a quote geriatric, type 1 diabetes pregnancy, and so I know there can be a lot of fear and uncertainty about what is healthy movement when you're pregnant. That is why I got trained in pre- and postnat al fitness, and this week on my podcast Project Swagger, I am sharing some key guidelines and the story of how I stayed active during my pregnancies. Listen now at Project Swagger . Support for the show comes from Net Suite. Every business is looking to see how AI can work for them. But while the possibilities are endless, the risks are as well, and you can't sit it out because your competitors are already making their move. But with NetSuite by Oracle, you can put AI to work today. NetSuite is a top-rated AI cloud ERP trusted by over 43,000 businesses. It's a unified suite that brings your financials, inventory, commerce, HR, and CRM into a single source of truth. That connected data is what makes your AI smarter. It knows how to intelligently automate routine tasks, how to deliver actionable insights, and can even help you cut costs and make fast AI-powered decisions with confidence , giving you total flexibility. From software and IT services to healthcare, equipment manufacturing, financial services, and many other great American industries, NetSuite delivers a customized solution for your business. If your revenues are at least in the seven figures, get our free business guide. Demystifying AI at netsuite.com slash on. The guide is free to you at netsuite.com slash on. Net suite.com slash on Buy Now Pay Later is everywhere. And honestly, we need to talk about it. Sixty percent of Coachella goers put their tickets on Buy Now PayLater this year. You can even pay off your DoorDash order and installments now, like your sushi. So this week on Networth and Chill, I'm breaking down exactly how these companies are making money off of you, why missing one payment can flip your interest rate from zero to nearly 36% overnight, and the uncomfortable truth about what it really means when you need to split a $200 purchase into four payments. Plus, I'm answering your questions, including what to do if you're already in over your head. Listen wherever you get your podcasts or watch on YouTube.com/slash your rich BFF . One of the things that's interesting, the parallel you make in the book and it's frequently made about the development of AIs to J. Robert Oppenheimer, as who we discussed earlier in the creation of the atomic bomb. It's one that demists and others in the field welcome, but Oppenheimer is also not just because I saw the Chris McNolan movie. I read a lot about Oppenheimer. He saw the bomb as an evil thing. It kicked off the nuclear arms race. He became a scientific exile during the Red Scare. How does he feel Demas feel about this? Because right now, as you s note correctly, there's a huge backlash to not just data centers in the US, but AI in general. The polling with the public is you know, only Trump has worse polls at this point. But AI is pretty disliked. So talk a little bit about that. Yeah, I mean I think he feels slightly that that's why you have to talk up the optimistic side of the AI story. I think he's a little frustrated with Dario, frankly, for going on TV and saying within five years 50% of entry-level jobs will be gone. Right. And it to propose to kind of say that kind of thing without proposing the policy remedy, you know, uh Demis sort of points slightly a finger at at Dario on that. But I think you know, I'm not sure I agree with him, frankly. I think, you know, Dario has a good case to say, hey, we've got to c call it like we see it, and and that will wake people up and politically meant maybe there'll be a stronger political Does he see himself as an Oppenheimer? I mean Oppenheimer went through a pretty rough period before we liked him again, right? Or when he got finally fed it at the end of his life. But there was a exile happening. There was exile happening. You know, I I think Demis embraces the kind of scientific glory, the leading of the Manhattan Project, the heroic story of you know, going off into the wilderness and focusing on nothing but the scientific mission. I mean there's something about Demis which is almost somebody called him a warrior monk and off he goes , you know, in his mind's eye to the desert in isolation. And that just is a super appealing self-image for him. Yeah, nothing narcissistic about that, but go ahead. That's that's the Oppenheimer bit that he wants to identify with. He doesn't identify or doesn't talk about, you know, the idea that he would be ostracized and, you know, there would be kind of a political Yeah, Dario's taking the flag for that at this point. Right. Yeah. And I think there's an interesting debate about whether, you know, if you look at Dario's principal stance on the Pentagon using AI for weapons, it was good in terms of raising public awareness of the issue. It didn't change the outcome. So if you're a demis and you look at that, you could say to yourself, well, it didn't change the outcome. So what was the point of that? I'm much smarter. I'm gonna go do this behind closed doors. I will talk to politicians as I did with Rishi Sunak and suggest things. They can take the credit. You know, what's the point of going public if you don't change how the world works? Mm-hmm. Absolutely. Except he didn't change how the world works either, but that's another issue. So we're in the middle of an explosion of growth around AI, and you write the estimates of achieving human level AGI, again, much disputed by 2030 now appear slightly conservative. Talk about how Demas looks at this because many I get a different answer, different people, and I don't think they even I think they're just guessing. I feel like at this point, they're just making it up. But talk a little bit about you get very different numbers from different people Aaron Powell You do get different numbers. I mean if you talk to people at Anthropic, they're really saying 2028. And what they say is that by 2028 there will be recursive self-improvement, meaning that the models will be coding the next model . And once they've done coding the next model, the new model will recode the next one after that. And so you'll get this vertical acceleration in the capacity of the systems. And at that point, you're done. At that point you're done, the race is over. And they say twenty twenty-eight in written material. I think actually privately. They even think it could be next year. So they are super near term in their prediction. Demis would you know, love to kind of nudge his prediction beyond twenty thirty, twenty thirty one, thirty two and as I was saying, that's partly 'cause his definition is, you know, can it be Einstein? Which is obviously the most expansive definition. Then there are people in between who say, well, it's about when most economically valuable human tasks that you could do in front of a screen could be taken over by AI. And then people might put, you know, tw aenty thirty kind of number on that. So I think that's that's that's the range. Right, right. And you just of course are seeing a lot of the layoffs from Meta and many others. So every episode we get a question from an outside expert, we have two for you. Here's the first one. Hi there. I'm Medina Friedman, the chair and CEO of NASDAQ. Based on all of the time that you spent with Demis , what do you see as the role of enterprise adoption in realizing the potential of AI? And if you do see it as being significant, what are the key obstacles that enterprises need to overcome in order to be able to achieve its potential within the enterprise? So I just referred to the layoffs. So go ahead. Yeah. So I think the revealing thing here is that the answer to the question of what does Demis have to say about enterprise adoption is zero. Nothing. And that shows us the separation between the builders of the models and the real-world users who are gonna actually put it, you know, change the way corporations function all that stuff. Although many of the builders are talking about it, th Anthropic would be the one who seems to be making the most headway in that area. Yeah. I mean I think Anthropic is very good and uh at at sort of going for enterprise tools. So coding first, cybersecurity second, and then having four deployed engineers, this is the kind of palantier model, who help corporations to adopt this stuff. So I agree with you. But certainly in a big company like Google, the people who are doing the Palantir stuff are in some other division of Google and Demis is really just focused on building the product. And and to the extent that you have the separation, and you're right, Cara, that it's not as clear in a smaller company like Anthropic, but I think the separation maybe does tell us something about what could go wrong here, right? That that you know one set of people are just focused on building something and they're not really thinking about how it's going to be used. Yeah. And then another is actually actively thinking. And we're starting to see more companies, for example, justify layoffs by pointing to AI. Sometimes that's not the case, but they still are using it as an excuse. Cisco laid off 4,000 people earlier this month while announcing record revenue and earnings. Meta laid off 8,000. Now, some of these, as I said, companies are using as a convenient scapegoat for too much hiring they did during COVID, which I think most people, especially the tech companies, did. Talk about n mass jobless ness and is that something that Demis thinks about? And if so, what's standing in the way of that? And he said that he thinks that companies who are placing developers with AI, quote, show a lack of imagination and a lack of understanding. Is he being again naive about corporate incentives around tech? And some people are moving too fast forward and it's not going to work, obviously. But talk a little bit about this. Aaron Powell Yeah I think there's sort of two extremes which are wrong. You know, one extreme is to say there's nothing to see here, there's not going to be any big jump in unemployment. We've been here with previous kinds of technologies and the labor market always adjusts and you know we, had the internet and actually unemployment was very low throughout that period. And I think that's naive because, you know, AI is not just another technology, it's more powerful, it's more general, it's directly competing, you know, with human cognition and it can be scaled if you have enough data centers you know up the wazoo so that you replace you know tons of people. So I think I think I think we can't just draw comfort and be complacent just because of the technology history. On the other hand, it's also way too simple to say now that this system can do all this stuff, you know, humans are done. Because, you know, new jobs will emerge to some extent. I mean, you know, it's partly that there are tasks that human humans are better at dealing with humans sometimes. So if you think about the the sales team, uh enterprise sales team in a big company where the job is basically to go schmooze the humans who are your customers and forge a bond with them. I I think humans have an edge at that. So I think there there are a lot of tasks where humans will remain superior. And to the extent that the economy grows because of AI, there'll be more of those tasks . So I think we should be ware of both extremes. I mean, one number I like. It's unclear. I I like to I like to cite this this this statistic which is that no technology has ever driven economic growth per capita at the frontier more than two and a half percent a year. So people who tell you that AI is gonna like double the size of the economy in fifteen years a smoking something? Oh abundance. You're not an abundance person, neither am I. The argument I'm so tired of that. The argument around rushing to develop, obviously AG, I think, has largely centered on beating China, but in many interviews with these people, it's I I call it the Xi or Me argument, like we gotta do it or China will. And I get it. But in a recent op-ed, and I really appreciated it that you wrote in the New York Times, you call for the US to negotiate a safety pact with China because we cannot beat them. And not not just falling behind, but there's certain things j as with nuclear energy, as with everything else. If we're falling behind , what incentive does China have in signing a safety pact? I think there's huge incentives for them to do so. And it's not just to keep them from being ahead of us, it's that there are commonalities here, as with, as I said, nuclear energy or nuclear weapons or cloning. There's a lot of stuff that is global. And oddly enough, I ran into Tony Blinken last night and he said I had I had urged him at the when he first got his job to do a global pact around safety and and with China especially. And he's like, Well, I didn't get that done. I'm like, No you didn't and then I walked away. I was rather intent on getting them to stop, you know, to pay attention to this. But talk a little bit about this, this sort of Xi or me and what happens with China, because you are correct, they are doing astonishing work here. And we had been ahead, obviously, the US or democracies, and now up with all democracies because Demis is in London. But talk a little bit about this race and also the need for a safety pack, which I think is critical. Yeah, yeah. So, you know, China does everything fast. So my book came out in China before it came out in the U S. And I went to China in in March for the launch. And I spent eight days basically going around talking to AI leaders, both in academic labs and in industrial tech companies. And what struck me was how often they brought up the topic of safety. And this was interesting because that is not what you hear from people who put the chip export controls in place in 2022 in the Biden time. So those people, you know, several of whom are my friends, and I've talked to them a lot about this, you know, they said to themselves, you know, credit to them, the scaling laws are real, AI is going to be super powerful. And they said this before ChatGPT came out. So we're going to do something, we're going to prevent the bad guys from getting this AI. But that definition of bad guy was China. And in my view, there's a more important set of bad guys who are criminals, who want to do cyber attacks, terrorists, individuals, rogue states, you know, random bad people. Rogues, just rogues, individual rogues. Exactly. And they missed out that whole category. And by making China the enemy and the rival and saying you cannot get AI and we're going to deny you the chips, and so you won't have NVIDIA, no NVIDIA, no no dice, you'll be left behind. You know, the mistake was to make China into the enemy and lose maybe a chance to talk to them about what about non-proliferation of this stuff? Maybe we should just say, you're a technology superpower, we are a technology superpower. The way we're going to avoid some catastrophic war between us is the same way we did it in the Cold War with the Soviets, which was parity, mutually assured destruction. Parity is actually good for balance and stability. Whereas when you're talking about random rogues using this stuff for bad ends, the Cold War lesson is that's where you need the IAEA to keep track of the material and the non-proliferation treaty. And my friends, you know, say who were in the Biden administration say, well, you know, yeah, but you can't talk to the Chinese because they don't care about safety. I went there and they did talk to me about safety. They say you can't trust the Chinese. And I go, you think it was easy to trust Khrushchev? You know, I mean this was the Soviet leader who banged his shoe on the table at the UN and put missiles in Cuba. Not an easy guy to talk to, but in that period is when we created the nonproliferation regime for that's right. And they have their interest of their economy being destroyed by a rogue something is just as high as ours. It's just really kind of kind of i i I I was always like, hmm, not so sure I want either of you. You know, I'd like a a global safety group that is that includes companies, includes legislators, includes China, includes the US. So that we have it's a very similar to nuclear exactly. We do we trust Khrushchev. That's a great way to put it. I I thought that piece was terrific. Now we have a second expert question for you. This is from Kent Walker, president of Global Affairs at Google and Alphabet and The Bad Cop, apparently. Let's listen. Hey Kira. Hey Sebastian. In reading the book, I was struck by how well Sebastian you captured Demis's vision of using AI as a way of solving intelligence to then solve everything else as a scientific tool to help us address some of our biggest problems. The challenge is that polling in the US suggests that America is one of the least optimistic countries about AI, much less optimistic than China is. Maybe we've had too many debates between AI accelerationists and AI doomers, but how would you suggest we go at the challenge of creating a grounded optimism , recognizing the challenges, but also encouraging people to put these tools to work to benefit themselves in their everyday lives and to benefit our society. Thanks. That is a great question. One, Google, you should stop being so aggressive, but go ahead. Yeah. So I mean we referred to this a bit before, but I think that medical breakthroughs that delivered AI-generated or AI-assisted drug discoveries that really save people's lives, that would be a great thing to change public opinion on whether AI is good or not. I also think that we ought to put in place preemptively before we totally need it, things like wage insurance , retraining schemes, active labor market schemes to help people get other jobs. Do that early. Don't wait around until it's obviously you know essential because at that point public opinion will be I think a useful statistic here is that in the twelve years from 1999 to 2011, the total number of job losses as a result of China renting the WTO was two million in the US. Two million is nothing. Two million is like the amount of labor market churn you get in an average month in the United States. And yet, you know, the backlash against China, the perception of the China shock, the fury about globalization politically was absolutely massive. So imagine if you got an AI shock that was way, way bigger than China. The reaction will be, you know, as you say, it'll be people in the streets, it you know, I mean, with the industrial revolution, what happened was you got revolution politically, you got Marxism, you got a whole lot of turmoil, and that was a revolution which you know took place probably one-tenth the speed of the AI revol ution and may arguably have been smaller. Absolutely. I think they don't even understand the anger, the rage that is building. And it's manifesting itself. I think a lot of the AI anger is a manifestation of affordability and nervousness about the future. And now it has become that. It has become that. You would put shares in AI companies or other stakes and distribute that democratically to all young Americans. Sounds good. No, I he doesn't seem interested in doing he does them, but he does other things more. I I that' thes stuff that he does populist, I'm not against like the Trump accounts or the I don't like him calling them Trump accounts or Trump RX or whatever. He has to put his fucking name on everything. But you know, but it that idea is correct. You know, UBS, some version of this or or more, like where are the jobs and do some really serious government studies and where we work with companies and you know this recent AI advisory board has nobody on it except for business people. And they could use critics, they could use someone like you, someone like me, someone like an academic, someone like you know, they just refuse to have m a m wide-ranging points of view here, that is probably going to be the biggest problem. I think I think it's also relevant to Kent's question that, you know, the record in the last couple of years, uh well, in during the Trump period, is that all the energy on legislation has come from the states. And the AI companies, the tech companies, have lobbied actively to stop that stuff from passing. And they have they have a decent argument in that a patchwork of state level stuff would be way less effective than a federal thing. But then that's why you need federal intervention, right? And which they are also trying to stop, which is why they're sitting in the front row of the Trump ignorance. Yeah, I don't quite I think that's a bit tough. I know you like to be tough, but I think that in the Ben Buchanan period when Ben Buchanan was the Tsar in the White House, what he said on the record publicly is that whenever he talked to the labs, they were willing to support what he was trying to do in terms of setting it up an AI safety institute, in terms of then requiring labs to share their models with the safety institute before they were released, you know, clearly that regul ation should have gone further. But he says that actually the labs were encouraging him to go down that path of regulating. And I think I I disbelieve that. But look with the Trump administration just as starting to announce now the same thing. The exact that they rejected. Correct? Now they do that with a lot of stuff, but their most recent noises has have been exactly what that Biden executive order said. Yeah, I agree. So so Trump Which is crazy. They're like, this is our new idea. And I'm like, well, that was two years ago, three years ago, Letchley Park. Completely agree with that. But that's a description of the Trump administration's U-turn. I think the position of the AI labs, which is what we were talking about earlier, I think they actually have been open to sensible federal action. They just hated the state action. Well they're getting the state action because initially they lobbied against any kind of tech regulation, right? And so they've they find them s they should they should be supporting it. They shouldn't have kneecapped Amy Klobichar, for example, or or they've been kneecapping politicians for a long time. And I think people don't trust them. I think that's where it is. And then when you have any manner of them swanning around looking like Daddy Warbox everywhere across the world. It's not great. It's not, you know, and Elon hasn't helped. Jeff Bezos certainly hasn't. And neither has Mark. I mean, I think the imagery. You don't have Demises there or, you know, Dario is a hero and like he's fine, but like that's it. Like that's the problem. There's no heroes. There's a lot of people who look like they're in it for the money. So so what is your explanation? Uh I'd love to get your view on this. So why is it that Demis is not particularly famous in the US. I mean I've been He doesn't wanna be. Yeah. He doesn't wanna be. Yeah. He could be. He could be all over the place. He could be d Dario Moting everything because he deserves that spot. I you know, I think he's a true wonk, probably. And so he's the science is everything. But you know, history is littered with people who the science is weren't as savvy. And Nikola Tesla wasn't as savvy as Edison. Edison was a PR person. He did all many of nefarious tricks against Tesla. Like, you know, I don't think he's that much of a victim the way Tesla certainly was if you read a lot about what happened there. But I think he likes to be pure. He likes to have that image of himself as pu that's my I don't know him very well, but that's what I can see. But he certainly deserves it. He certainly deserves the attention. You know, Jeffrey Hinton's likes to take up space, that's for sure, but you could see him out there more. But no, here he is with your book. So obviously he's talking to you. So last question. Today's AI race is exactly the scenario he had hoped to avoid, but in an excerpt in The Atlantic, you wrote that Demis has, quote, come to see salvation paradoxically in his own career advancement and securing personal influence. We have concentrated so much power in the hands of just a few people. After years of talking to many of them for the book, what do you make of his conclusion and what does it mean for the rest of us who are worried about concentrated power? I know I am, of anyone, even if they were the l angels. I don't feel like they should have this many small amount of individuals to have this much power over something so important. Of course, it's happened in history, but it's never ended up well. And this is quantumly more powerful than any like railroads, trains, the telegraph, st uff like that. Are you worried and should the rest of us be worried about this concentration of power, even if it's in the hands of people that are better than others, but a lot of them aren't? Yeah. So Yeah, yeah. Yeah. Look, I think in a way the story of Demistisabas is, you know, of a decent person, a good person who wants to be good but can't actually have a good impact because he's inside this race dynamic, inside a big corporation, and whatever he may personally think about safety, there are these larger forces that are driving him on. And if he were to quit tomorrow and take a professorship in theoretical physics at Printon, it wouldn't change the race, it wouldn't make the world better, it wouldn't do anything very constructive. I'm not sure I want him to do that because you know, it would just be a gesture and it preferred to focus on the outcomes. So what this points to is that we do need government intervention. You know, the The only way to counter the power of this handful of individuals who are kind of collectively very powerful but individually impotent in terms of the big social questions, you know, you need government to take a serious intervention and you know why not? We have drugs in the United States which are approved by a regulator and if they're dangerous, they're not allowed to be released, right? Why would an AI model be released without being tested by the government first? Why would we allow, you know, open weight models that can be ripped off by anybody, abused by anybody, you Right. So where does that leave the rest of us if AA goes the way that social media did and the government fails to act? I mean, obviously different governments are doing different things, but there's no coordinated global effort at all. But y and in this country, forget it so far. And of course we did nothing to s to slow down the the relentless and often poisonous and toxic pace of social media. So where does what has to happen? What do you see happening? What does Demas see happening? I think I think he's got to the point where he's trying a little bit to you know do politicking behind the scenes and suggest that you know more action is needed and it has to be collective and the government needs to lead that. But whether he's putting real energy into it, I can't tell you for sure. And so I think he's just , you know, that history we talked about before of Project Mario and trying to get safety It's almost like he's burnt out on trying to solve the governance problem. And so he's just heads down on the technical problems, which is a regrettable place to end up. But you know, all I can say is that, you know, I mean I guess the most optimistic thing that's happened in the last few months was when Anthropic released Mythos and the Trump administration freaked out. And they called in the heads of the banks and said, You guys are gonna have hacks that empty your bank accounts unless you can harden your cyber defences