The Quest for Superintelligence and Safety

Exchanges on the MNA and IPO landscape. Exchanges on the dynamics affecting global trade. For the sharpest analysis on finance, business, and the economy, Count on Exchanges, The Goldman Sachs podcast. Listen now This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a fifty page restoration block or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online makes sense? There's no place like Chrome. Check responssees setup required compatibility and availability varies eighteen plus When you need to build up your team to handle the growing chaos at work, use Indeed sponsored jobs. It gives your job post the boost it needs to be seen and helps reach people with the right skills, certifications, and more. Spend less time searching and more time actually interviewing candidates who check all your boxes. Listeners of this show will get a seventy five dollars sponsored job credit at indeed dot com slash podcast That's indeed d. com slash podcast. Terms and conditions apppply. Ne a hiring hero? This is a job for indeed sponsored jobs. Welcome to Profty Markets Cracks are forming in the open AI story. Last week, the company reportedly proposed giving the US government a five percent stake worth roughly forty three billion dollars as a way to share the upside of AI with the public. Critics, however, argue it would amount to a government bailout and see it as a troubling signal for both open AI and the broader AI boom. news came afterfter reports that OpenAI had pushed back its IPO plans until twenty twenty seven. adding to concerns about the company's financial position. So we wanted to speak was someone who has spent years studying the history of AI. And who also believes that openen AI could run out of money in the near future. Sebastian Mallaby is a prominent journalist, author Pulletz surprze finalists and senior fellow at the Council on foreign relations and today He's joining us to discuss what is next for open AI, what is next for the AI industry and what investors should be watching. Sebastian, thank you so much for joining me on the show I'd love to start with An article you wrote back in January. that was titled This is what convinced me OenAI will run out of money And you said back then, quote, my bet is that over the next eighteen months, openpen Eye runs out of money we've been seeing A lot of red flags since then, the delaying of the IPO later we saw this proposal for the U.S. government to take a stake in the company Um, I guess I'll just start with what do you make of the recent news and do you hold to your prediction? Yeah, I do hold to my prediction. back in January the burn rate was just crazy so that although open AI had products and quite a lot of traction, nine hundred million consumers They won't be able to charge money for the product, like five percent of the retail consumers were actually paying if you look at a chart of where these users are You know, the US is the number two market, India is first and the next three are kind of Brazil, Indonesia and so forth. So these are not rich consumers. You can't charge them very much money And so they had a business model that imagined that they could throw money in all directions you know, a collaboration with Johnny I to have a new form factor which would supplant the iPhone Sving stora, video generation models and all this stuff, All of which is very expensive And yet the revenue side simply wasn't there. So the burn seem to me to be totally unsustainable. And even though Sam Altman is a magician when it comes to raising money He wasn't going to be raising six hundred sixty billion dollars, which is what the internally projected burner rate was for the next five years when you looked at the documents back in January. Now since then, What's happened is some good news, right? becausecause OpenAI, I think has recognized that it had to get the burn rate down. It's pulled out of a bunch of data center building products, know Stargate, all that stuff. Ccel Sora, the video generation model, which was a total money loser and it's tried to impose some sort of order on the chaotic management. But it's only been kind of half successful. And in the meantime Open AI is squeezed between anthropic, which is much better at the frontier enterprise applications like you know, coding assistance and cybersecurity stuff and agentic stuff. And then on the other hand, it's squeezed by The Gemini model from Google Deepmind which has now reached more retail consumers and is way better at monetizing from that because Google has plugged AI into its search advertising business and that business is now doing more revenue than ever So I just think that, you know, open AI is technically a good lab But it's very hard to monetize when you have a product where there's a lot of competition and it's kind of a commodity and they're not terribly well managed and they've relied too much on the fake it till you make it sort ofan valy tactic of kind of Weird smmoke and mirrors fundraising gambs. If you look at the fundraising they did and announced earlier this year, the headline number they raised was one hundred and twenty two. billion dollars which is an astronomical amount But when you dig in and I'm amazed the press didn't point this out more About two thirds of that amount was kind of promises in the future, conditional upon having a successful IPO. payment in kind, like, you access to compute. The actual real money was a small share of the total fundraise, which raises the question, why announce This massive one hundred twenty two billion headline number when anyone who digs into it can see it's rubbish Well, the answer is they're trying to head fake investors into putting more money in try to persuade people they have momentum. they don't And this news that you pointed out just recently that they have delayed seems their IPO into next year, is just the latest icing on the cake, the latest evidence that They talk a big game, but they are behind where they say they are Do you think that the delay of the IPO was in large part because of all of this because perhaps Sam Altman and the company know that as soon as Wall Street actually gets like an audited review of their financial statements then suddenly The tide will turn on this company and suddenly people will say, sure, you might have a great product, but this is not a sustainable business model. Do you think that That was the concern that people might actually see how the company actually works N hundred percent, I mean everybody remembers the WeW workk story When we work was this, you know rocket ship back in like twenty nineteen and it went out with a prospectus to do the IPO and people looked at it and said, this is a joke And nobody wanted to buy the shares and Po never happened so you know, you can fail in going for the IPO and open AI is in this very tough position where On the one hand, it needs the IPO. because it can't hope to raise enough money if it stays private. On the other hand, if it tries to do the IPO It may not succeed and then it's really cooked Just looking at how much they are spending at the moment. could we just saw the financials that were released by Ed Zitrin who is this independent journalist. he got his hands on the numbers. They generated thirteen billion dollars last year in revenue They spent thirty four billion dollars. which means that their operating loss was twenty one billion dollars I mean, we could talk about the net loss, which was even higher than that than that number, but that seems to be like a good roughly estimate of how this business is actually doing And you mentioned that they're stuck between on one hand Gemini and then also anthropic Anthropic is an interesting one because We also don't know much about that business And we know that they're unprofitable and we know that they're in a similar business to Oen AI. And as you say, this technology is becoming increasingly commoditized. They said there were reports that maybe they were coming up on a quarter of operating profitability, but I think we probably have to take that with a grain of salt because we don't know how they're doing their accounting My question How does anthropic compare to open eye from a business model perspective. I think you're making a good point and I agree with you that we don't know as much as we would do if Anthropic were a public company or if the prospectus was public. What we do know is that Anthropic has always targeted enterprise customers which means the type of customer that actually pay for the product And we know that it's been ahead on stuff like Coding assistance and cybersecurity AI. Not that open AI is bad, by the way. I mean, it's not far behind. But I think anthropic is the cutting edge on those particular applications that Enerprises are really willing to pay for And meanwhile Anthropic has not been sort are distracted into announcing a whole suite of retail oriented business initiatives, which came to nothing. I mean, OpenAI announced it was going to do shopping at one point and that doesn't seem to have happened. It said it was going to do ads. I'm not sure they've got terribly far with the ads. It did generate the you know Sor a video model, which just was a huge money loser whereas Anthropic never went down that path. So I think anthropic ' been way more laser focused on the part of the market that makes sense, which is the enterprise part and and just better managed. The other point I'd make is that Anthropic amongst all the frontier labs is known as the one where the Chan in terms of the scientists is the lowest. People go there They believe in the mission, they believe in Dario Amade as the leader And they tend to stay there, they don't hop around Whereas all the other lads are subject to job hopping and that's obviously disruptive I think one of the questions that is in investors' minds, especially if you're worried about the potential of an AI bubble and the potential that and AI bubble might pop is is it an openI problem or is it an AI problem? Is it that open AI is just bad at managing their finances and they pursued all these side projects and they don't really know how to get their spending under control Or is AI as a business model just too expensive relative to the amount of revenue that could be generated by charging customers for using Chge GPT, or charging enterprises for these larger enterprise wide AI contracts What is your view on that debate. and just for context for our listeners, you wrote Power Law, which is one of the most famous books ever on Venture Capital. and it's kind of about how venture works as a business model where you do lose money for a number of years, and then you figure it out eventually Is AI going to be that story or is this different? My view is that we have an open AI bubble, but not a general AI bubble So I think open AI for the reasons we've discussed this is a fifty fifty., it might work. I'm not saying I don't know that they're going to fail. I'm just saying there's a fifty percent chance that by next summer will find They couldn't really go public in the private markets, they can't raise enough money and they have to sort of sell themselves. at some sort of discount through another company. it could be Amazon or Microsoft or some other company that wants an AI team because technically, open AI is a good team Right. Now on the more general issue Yeah, there's debate at the moment about whether enterprise customers are having a oh my go moment where they think, oh, these tokens are just so expensive now I have spent the last eighteen months telling my teams that they should just go out and go wild with AI and experiment and do whatever they feel like and token max and the more tokens you use, the better of an employee you are because you're showing that you're AI forward. And now wow, this is expensive and I've han't seen a productivity game yet And so what am I doing here? I have to rationalize this. and There are lots of stories out there about how companies are imposing a sort of middle layer between the user in the enterprise and the models. And the middle layer is there to switch a query So that know, if it's a simple query that I'm asking, it gets routed to a cheap low token consumption model. And then only if it's a seriously difficult one will it go to a fable or something expensive. So I think there's some sort of sensible rationalization about how the AI customers are spending money on this technology. But fundamentally Fundamentally, if you look back at what's happened since the release of ChatTPT, the clear story is This is unbelievably exciting fast progress I mean When Chatubiti came out, the thing hallucinated nonstop And thenen when GPD four was plugged in six months later, it basically stopped like eighty percent of the hallucination Then you've got very long context windows, so you could put a whole toull story novel into the model and then query it Then you got, you know these reasoning systems that could do math and logic, which had been impossible before. Then you get aentic system, then you get Coding assistance, then you get cybersecurity systems. Now you've got like bespoke AI autonomous scientists emerging This is unbelievably fast progress So I fundamentally think that, you know, AI as a sector And therefore, the demand for the semiconductors, the data center businesses, all these things that people worry about I don't think that's a bubble. I think that's for real. and it's going to take a little bit of time companies to figure out how to ration their people's use of tokens so it's sort of sensible Basically they're going to consume a lot of tokens. another data point that bears might present that we saw last week is Meta launching their cloud business And this would be the argument against what you're saying, which by the way, I agree with, but I want to play Devil's addvocate You know, The very thing that meta said they wouldn't do they're now doing. They said that they would only launch a cloud business if they had, quote, overbilt. This these were Mark Zuckerbgs words just a few months ago, the plan was let's build out all of these data centers, build out all of this compute capacity because we within the meta organization need it so desperately because we're going to build all of these internal AI products and we're going to AI turbo chararge our business And then they turn around and say, actually, we don't have the demand internally that we thought we did. u soil we're going to sell it to someone else and we're going to let someone else figure out how to sell an AI product and how to make that a profitable business which seemeems quite bearish from a bubble perspective because it basically said, I mean, who else but meta? be the one build out their own suite of AI products. If Meta canan't crack it, if open AI is struggling to crack it, TBD on anthropics then Who's going to crack this? Who's going to make this not just an interesting technology, but an interesting technology that makes money And we've seen the same with SpaceX, of course, that they also decided to sell their compute capacity to anthropic and others, because XAI, their own model hasn't really got much of a uptake And so they don't need all the compute they've built for their own model, Therefore they're selling it to others. So You could view this as a bear signal, as you've just described, or you could view it as a buill signal because it means that you've got some consolidation going on in the frontier model space and less competition. means better margins for the remaining participants. It means that maybe there will be more pricing power for the ones that are less standing So I find I don't agree that's I think that's the proper reading. The proper reading is we have a rationalization of the market. If you looked at the sort of US ecosystem you know three or four months ago, you had XAI trying to compete, meta trying to compete. And then on top of that, you had the big three Google deep mind open AI and anthropics. So that's five. and that's before you count Mistrad in France, cohere in Canada and all the Chinese models, right? has a lot of competition and I don't think that this thing is going to consolidate down to a winner takes h allle sort of you know nineteen sorry, twenty ten's social media platform or something But I think some consolidation is in order such that it looks like computing where there's kind of three or four big providers. So now we've got, you know three leaders who are still standing within the U.S, plus the foreign ones That feels good to me in terms of the future business stability of the sector. If Op AI runs out of money as you say per your prediction. Um What do you think the outcome would be? I mean, one of the things that you wrote is that maybe it would be absorbed by another company Um, I mean, how does that play out? if indeed What you're saying might happen does happen So look, I think you, we've seen lots of examples of either acquisitions or more recently Aqua hires where you know you have a smallish AI company inflection, which Mafa Sonyan was running and then it got sort of sucked into Microsoft. or like character AI, which got sucked back into Google So there's a playbook here. now open AI is a lot bigger than either of those two So it would be a more complex playbook. but basically it seems to me that Um, you know the demand for AI talent And for AI products And therefore the compute infrastructure under that serves that demand I don't think that's going away because fundamentally I think this is useful stuff that people are going to figure out how to use Proively And so I don't know whether the whole of open AI gets bought by Amazon or Microsoft or some other requirer, or alternatively there's some kind of fancy Aqua Hire deal where part of open AI is sucked into a big company, or alternatively that like You know, there's a bit of a splintering and the staff, the technical staff at open AI individually hired into other labs. Wh knows, right? what I'm saying is that there's a fundamental problem with the way they're going about their business model. I think they understand that, which is why you know they pulled out of data center building and various other things in the last six months But they've got some way to go to fix things and patch it up. And you know, one of the lessons about how you do startups um, you know, coming out in my previous book, The Power Lw. is that when you have a very high valuation Down round is super painful you know, they're valued In the last round at eight hundred fifty two million billion dollars post money. and in the secondary market they're trading for a lot less than that. and if they were to sort of Just say, okay, we accept. We're really worth six hundred billion Um, you know, the hit to everybody's equity options Inside open AI would be horrible and they would lose people And the hit to investors who had believed in open AI would be bad and they would get pissed off. and the whole momentum machine that Sam Alman has built. Ready through a convulsion. Now, it might be what you have to do to make this thing sustainable because But my point is once you ratch it all the way up to this very high valuation 's difficult to climb down Um and and that is why I think he says Why doesn't the government have five percent because a strategy to get out of this box that he's in is for Altman to give five percent to the government and then the government will say, right, you know, open AI is too important to fail now because we own five percent or ten percent or something, and they'll do what they did with Intel, which they took a ten percent stake in last year. And next thing you know, the commmerce Secretary Lutnk is like calling other tech companies in the valley saying, you're going to do a deal with Intel. You're going to bring Intel in as a partner on your next project, blah blah blah. And so you know, you've got the U.S. government Trumpy US. government, strong arming other companies into giving business once they're in your corner. So that I think that is what Sam Amanans strategy is here to kind of recruit the, you know, the investment banker to whom you can't say no the U.S govern whichich seems like he's basically just trying to take some sort of work around shortcut around capitalism. And it seems like we are increasingly seeing that. Like if you can't figure it out in the free market thenen, oh, let's just go over to Washington walking in the White House kiss the president's feet and then hopefully he'll save us. And we are increasingly seeing that that is what is actually happening. We're seeing the government taking up stakes in multiple companies We're seeing the odds that the government will take stakes in even more companies. Those are going up They may indeed take a stake in opening e. last I checked on the prediction markets. The odds of that happening were more than a third Um, It's possible that they would do the same with Anthropic, with Palantir, with Andreo makes me very upset because I think of it as cheating I think that you're kind of cheating the game of capitalis, and I'd be curious to get your views there And then following up on that if that actually happens say openp A is running out of money. And then Trump just Bails them out in whatever way, we use taxpayer dollars to just continontue to subsidize the business What comes after that Does that mean that open air is fine? Does that mean that The rest of the AI industry is on shaky ground. I'm not even I'm not quite sure how to even model out potential scenario First of all, I think your formulation that they're cheating capitalism and know they're going to the government and doing an end round around capitalism I mean, I think that's a good perceptive and quite amusing insight. So thank you for that. I also though would say that you know this is like just the way the world is going, I mean or at least the U.S is going. So if you look at the number of American companies in which the U.S government has announced either done a deal or has announced the deal and it's yet to be consummated You know, a colleague of mine called Jonathan Hillman at the Council of Fign Relations did a formal count which just went up on the Council of Foreign Relations website. and the answer is there are thirty of them thirty such companies since the Trump team came into power in january twenty twenty five. where where there's an equity stake from the U.S. government and a private company. So this is where the world is going. And I think this trend has been very much encouraged by the deceptive example of Intel Right. So in the case of Intel, if you look at what the performance has been, Since the government took a state last August, it's been fantastic. I mean, it's been way better than the Philadelphia semiconductor index, which is the normal index you would look at as a kind of comparable for how Intel is done. Intel, I think, is up like almost four hundred percent. The socks or the semiconductor index in Philadelphia. that's up like one hundred fifty percent So these Intel has done incredibly well since the government came in And I think people just lose sight of the fact that you know, yeah, it did well becausecause you've got, you know, the commerce department calling up other companies and ordering them to do business with Intel. So Intel gets a whole bunch of contracts and it's like turning its game around Puse You've got the government behind, you know, picking a winner. now It's one thing to say the government might have a justification picking a winner when we have a problem with all of the cutting edge semiconductors being made in Taiwan. We don't want to be reliant on an island that could be invaded by China. And so we want domestic US semiconductor manufacturing. I get that argument, right believe in extending the same argument to open AI which is just one of multiple American foundoundation model builders. We don't need open AI for any strategic reason, right? So there would be no justification for picking a winner around open AI. So I think it' I think that the you capitalism is sometimes justifiably twisted because you have a national security reason to do so Tcking open AI would not be a justifiable instance. Well, I could imagine that the justification that would be floated is open AI isn't systemic to the real economy, but they maybe try to say that, but it's systemic to the stock market because You know, Microsoft's future revenues depend so heavily on open AI Uh so do I mean, I mean Google, Amazon, Excel, I mean, all of basically all the hypers scans, Oracle. A lot of these companies are very, very important. to portfolios. they are what make wealthy people, wealthy in a lot of cases. And maybe the argument for Trump would be Oh well we We need to keep this thing afloat, otherwise people's stocks are going to go down What would you make of that argument? I'd say welcome to China. I mean, that's the kind of thing the Chinese government would do is prop up the stock market with government intervention of that sort. I mean, look, in the United States, when the Federal Reserve you know operates a policy that looks like it might be about stabilizing the stock market. People freak out and say, well, that's a Fed put. And you know, that creates bubbles, more bubbles in the future. And you know capitalism doesn't work unless there's real risk involved. and that's the Fed If you have like bunches political types in Washington, you know, the Cmerce Department and so forth picking winners and distorting outcomes in the market, you don't have a market anymore. It's not a free market. Your point about this is an end run around you know, capitalism or to say the same point differently You know, this is an end run against the notion of a fair level playing field on which different companies compete fairly and then the most efficient ones win That's what we're supposed to believe in as the wellspring of efficiency in American capitalism. If you start Dleveling the playing field by picking open AI as a winner. You've just Trust that. We'll be right back after the break. And if you're enjoying the show so far, send it to a friend and please follow us on YouTube, Spotify, wherever you get your podcasts upport for the show comes from Delete Me The internet makes it easy for scammers, stalkers and hackers to do their thing. That's just a fact But it's important to remember that there are ways to protect yourself That starts with protecting your data And Delete Me can help you do just that Delleeme does all the hard work to wipe your personal information from data broker websites. It was named the top pick for data rememoval serervices by Wirecutter And our producer Jennifer Sanchez has triried to delete me. She was surprised to find out how much personal information was actually already out there. but totely made it very easy her to clean it up. takeake control of your data and keep your private life private by signing up for Delete M Now at a special discount for our listeners. Get twenty percent off of your deletely plan. When you go to join deeleteme d. com slash prof G. and use promo code profG at checkout. The only way to get twenty percent off is to go to join deeleteme d. com slash profG and enter code profG at checkout. That's join delete mee d. com slash profG code profG Your summer weekends fill up fast, but Crocx has your back. Road trips, beach days, last minute getaways, whatever's on the agenda, swing by your local store and find your new goat too. Try it, style it, make it yours. becausecause the right pair doesn't just show up, it shows off Walk out ready for whatever's next. Visit your nearest crox store today When it's time to scale your business, it's time for Shopify get everything you need to grow the way you want All the way Stack more sales with the best converting checkout on the planet. Track your chihings from every channel, ride in one spot, and turn real time reporting into big time opportunities. Take your business to a whole new level. Switch to Shopify. Start your free trial today We're back with prorofperty markets This is a good segue into China, which is a topic that you also wrote about recently the title of your piece was quote I Went to China to see its progress on AI. We can't beat it. I was in the New York Times recently, you're up ed What did you learn about Chinese progress on AI and why do you think that we can't beat to. We haven't mentioned this yet, but I'm going to mention it now because you've given me the excuse. So I published a book this year called the Infinity machine about Demis Sasabis and I was going to get to her But there you are. I'm glad you mentioned it. Go read the infinity machine folks, but seriously. The thing about China is it does everything faster And so they published, although they got my manuscript last, and then they had to translate it into Chinese and then they wanted photographs and other embellishments produce in a way a more complex product, but they actually published it before Penguin press in the United States or any of the other deals I had in other countries. So I go to China right at the beginning of my book tour, I spent eight days you know going to four different cities, Hangzhou, Shenzhen you know, Shanghai, Beijing, talking to both computer scientists at the private labs like You know Huawei and annt group and so forth and then also talking to academic computer scientists from universities. And what struck me about these guys is that first of all They talk about safety They just bring it up. The notion, which I've heard from friends in Washington that the Chinese don't give a damn about AI safety is just not true. They do talk about safety now. I'm not claiming government's policy is to pursue safety or that the majority view in China is that they want safety. China is like the US. China has some accelerationists and some people who want to go slower because they're worried about the safety issue. That's the same as the US. So neither side is going to de escalate and start going slower unless the other one does as well What I'm saying is Caricature China is like only acceleration is one hundred percent. That's just wrong And so there is scope to talk to them about safety and maybe we'll come onto that. But the other thing which I observed is that China is very good and very focused on applications And so if you go to a company like, you know, Hyike Vision which is under U. S. sanctions, and it's kind of an out of body sort of double take experience when you go there because On the one hand, it feels like an American tech company. I love tech companies. They're kind of all about building cool things and making the world better. I kind of buy that. I drink that cool aid, I kind of believe in it. I like technology, right? So I see these people trying to build cool technology and they show me stuff like, for example There is an AI kind of scanning camera thing and you point it at some water and you get a reading on the pollution count in the water. And because they've created that Guess what? There is an internal market in water pollution reduction between different Chinese cities. So if you're the downstream city You will pay the upstream city. Rduce the pollution in the water that's going to come downstream to you And so you can do pollution reduction when you can measure the pollution. And this is what they're doing at this company, this is what they're building. They're also under sanctions these guys by the US because the US says and you historically this was actually true, that they're building other kinds of cameras which are good for surveillance of civilians and so forth in Xinjiang and whatever So So they're both bad guys and they're cool guys. It's a difficult thing to figure out. But whatever they think, whether they are bad or cool, they ain't going away These guys are for real. They are building cool technology. You go to Huawei, they've got application after application. You know, here is our special you know AI to service the bullet train between Shanghai and Beijing every evening. We used to have human technicians mechanics who would go under the train and make sure it's all fine. Now we just have AI cameras and a couple of robots and they fix the train for you They are doing this. We're not stopping them. We have imposed chip export controls on China to try to hold them back It hasn't worked. These guys are moving ahead. And the latest thing as you probably saw is this model from a group called Jipu. in China, which isn't quite as good as mythos fromanthropic, but it's pretty close So we are kidding ourselves if we kind of assume away the reality of China being a technology superpower. And we need to On the contrary, get our heads out of the ostrich position in the sand and start talking to China about what happens when they have a Mythos level model which could hack every single bank. in the global financial system and wreak havoc We need to persuade them not to release it on an open source, open weight basis Beacause then any criminal can do it and they won't be an off switch. What is your view then on AI policy with China? Obviously the big debate is should we have these export controls? Should we sell chips to China? Are we selling weapons to our enemy? Or do we need to sell dumber chips, basically dumber weapons to the enemy? or should we not have these export controls at all I mean, do you think that we should have a policy or is the path forward more of a metethod of diplomacy I believe in American power first of all. I work at the Council on Foreign Relations in New York. And you know, we do geopolitics all day long and I believe that US power is generally a force for good. So I would rather The Chinese were behind on AI, okay. And so to the extent that a chip export ban s us to be ahead. I support it, and indeed, when it was first announced, In twenty twenty two, I wrote a massive long essay in the Washington Post about why this was a good idea The reason I've had my diets recently is that I look at the results and I'm not seeing that Chinese models are that far behind. And in the meantime, because they're not far behind, I think we have to reckon with the reality that they are building models which are going to destabilize the global cyber system And unless we persuade them. not to release them on an open weight basis, which is what they're doing at the moment We have serious trouble on our hands, like everything in cyberspace will be destabilized. And we need a policy to deal with this proliferation risk. and I would be willing Now I'm in favor of the chip export ban if we could have it for free and there'd be no downside But if the effect of having chip export controls is that we can't talk to them about an agreement on not doing open weight mythos Then I'm willing to trade a bit on the Cipbackport band Just looking at some of the how these models have affected the ecosystem. and something we were saying earlier in the U.S. you've got Anthropic, you've got Op AI, you've got Gemini. Those are kind of the heavyweights. in the U. S right now, but it does seem as pricing becomes more of an issue companies are more interested in cheaper models, which usually means Chinese models and indeed That is exactly what we're seeing when we look at U Open router, which is basically a tracks developer marketplace for AI models. Chinese models went from less than a third of developer traffic in late twenty twenty five to sixty percent. By mid twenty twenty six, there are some companies that American companies that have started using Chinese models Csa, Airbnb Shopify, Uber, Microsoft is currently testing Deepsek U what do you make of this transition over to the Chinese models, particifularly the cheap Chinese models, and what role does that play in potentially a policy discussion Well, I mean, it shows you that they make good models that serious American companies are thinking of using And so that's another argument for why You can't just pretend that China can beaten and that's the end of it. And these guys are for real and we have to work with them, not just against them U no, I think it's useful to just for a moment, think through the lens of the Cold War where when there were nuclear weapons In the Cold War, there were two kinds of big risk, right? One was a nuclear conflagration between the Soviet Union and the United States And the way we prevented that was through mutually assured destruction, basically close to parity in the power of the two arsenals and therefore deterrence On the other hand, there was a different category of risk from nuclear weapons, which was the proliferation of these systems to rogue states or terrorists and so forth And we dealt with that with a separate mechanism. which was the non proroliferation regime Now, the point is we were both competing with Russia haaving an arms race with Russia, having a Cuban missile crisis with Russia, being told by the Russians that at the United Nations We will bury you, as Kristchev said, when he banged his shoe on the table. there deeadly serious competition between the two superpowers, but at the same time there was cooperation. on a non proliferation agreement. The way I see the future with AI is that we'll do the same We will have inevitable you know, competition between China and the US But we also, I hope, have collaboration because the proliferation risk is too awful to contemplate unless you have some collaboration It seems though that what they're doing is basically stealing what we have. People are calling it distillation And you wrote about this and your definition of distillation, quote, Every time a US. lab produces a cutting edge model, Chinese rivals quickly reverse, engineer its capabilities and build a copycat version, the follower has the advantage And when I look at how mean, Companies are switching to Chinese models because Chinese models are cheaper And as you say, maybe they'll use the more advanced cutting edge models in America that are more expensive for certain tasks trying these models for other tasks whichich essentially means that we are kind of maybe we're collaborating, but also you could say that we're sort of ceding advantage to the enemy, to the Chinese players in the AI ecosystem. And it seems that the reason that those models are good is because of distillation, e Sft Um I don't know if I'm being crude by calling it theft. I don't think I am. and I think The Chinese have shown a pretty strong track record of stealing intellectual property from the US and then going out and monetizing it on their own terms. I mean, what do we know about this process of distillation and what do we know about Why and how? The Chinese models have gotten so cheap and therefore so successful on a global scale. So distillation is a process which involves asking a very strong model, like a new American model comes out The Chinese copycat would ask a ton of questions to that model. The answers And the answer is amount of training data such that you you can train the Chinese model like if the question is like this, the answer should be like that. and When a frontier, like a first mover, an American lab has to train the model in some specific, very complicated frontier expertise like let's say, you know quantum physics They expensively hire a bunch of quantum physicists and engage them in creating problem sets and know, model questions and answers, and generating that training data for the AI is a super expensive, time consuming, painful process But if once you've created the AI that can Acate all those quantum physicists The Chinese can come along and not hire a human quantum physicist, but just query the machine equivalent And that's what distillation is. Now, when these Chinese companies do this It is not illegal But it is against contract. In other words you when you sign up to use an American model, you sign you check some boxes and you sign an agreement saying, you know, I'm not going to like query you Gazillion times and then train my own model by copying what you've done And so they are violating contract. but not sort of federal law. That's my understanding of it Whatever the legal niceties? The question is, can you stop it? I mean, I'm all in favor of stopping that, if we can And it seems to me that Anthropic and you know Google and Ain AI have all of the commercial incentives in the world to put in anti distillation safeguards if they can come up with something. So I think this is a like a self solving problem Insof far as it has a solution And by the way, I should add, you know, Elon Musk the other day or a few months ago casually admitted that His company, XAI, had distilled from one of the U. SS frontier competitors So it's not just the Chinese who do this this is the rough and tumble of the marketplace It's not nice But the practical question is, you know, let's stop it if we can, but insofar as we can't We have to live with a reality on the ground, which is that the Chinese models are good Something I can't figure out is I mean, if these AI labs are as capable as they say they are, can they not figure out some cybersecurity method to stop the distillation from happening? Mythos is the most powerful cybersecurity technology and software that the world has ever seen, but we can't figure out how to get these Chinese developers to stop querying and replicating the same software. I'm sort of like surely you guys can figure it out. I guess my fault would be say they do figure it out, sayay we do put an end to Chinese distillation of US AI Would that not One solve America's problems in one fell swoop and two, kindind of put an end to Chinese AI or at least the progress that they have been making I mean, is that not kind of a poison pill for for China I'm not sure is the answer. whether if you could stop distillation, and by the way, I think the latest anthropic models do have some anti distillation technology built into them. So we'll see how effective that turns out to be. It's going to be obviously a You know Cat and Mouse both sides trying to get smarter on this one. But to answer your question, let's posit that U. S. labs figure out a way to stop destination would the Chinese fall behind Like a lot just a bit I'm not sure anybody really knows the answer I kind of suspect that You know, if they needed to generate their own data, they would and they would pay more money and it would be more expensive and it would take them a bit more time, but they would get there because they've got plenty of extxtremely smart Chinese scientists that they could engage in generating training data We'll be right back. And for even more markets content, sign up for our newsletter at proftymarkets. com wishing you could be there live for the big game, soaking up the atmosphere of the crowd Too often, life gets busy orr the price hld you back Priceeline is here to help you make it happen With millions of deals on flights, hotels, and rental cars, you can go see the game live Don't just dream about the trip. Book it with Priceline. Download the prriceline app or visit priceline. comot Actual prices may vary limited time offer This episode is brought to you by Street Easy. Here's the thing. Wna be the grandparents who bolted to the Burbs or the cool relatives still living in NYC? The city that people come to to be at the center of everything. With twenty years at NYC know how, Street Easy has the tools, agents, and guidance to make you a forever New Yorker too Visit streeteasy. com to buy a rent in NYC. StreetEasy is an assumed name of Zillow Inc, which has licenses in all fifty states. Flies crash in the cookout. Take back your space with STEM's portable flying insect fan trap. Set it down wherever flying insects show up and let it get to work. Using UV light in a fan, it attracts and traps bugs twenty four seven, like house flies, fruit flies, gnats, and mosquitoes. It's easy, mest free, and people and pet friendly when use as directed Live life never buuged with STEM. Visit stTEMforbugs. com to learn more We're back with proroperty markets Okay, let's turn to the book for a moment. Your most recent book was The Infinity Machine Demis Hassabis Steepmind and the Quest for suuperintelligence. Um there was one quote from the book that really stood out. you said quote, If you couldn't negotiate safety mechanisms inside one company, what chance would there be to negotiate common safeguards among multiple labs in multiple countries, which really relates to kind of what we're discussing here in terms of AI safety and AI policy But also there's an important implication to that, which is that Safety mechanisms were not able to be negotiated within a company, What did you learn about the inner workings of these AI labs? And why can't they figure that stuff out Embedded in the story of Google Deepmind and Demisisabis is this sort of morality tale about somebody who really wanted to make AI safe and that was his sort of driving passion. From the time he founded Deep Mine in twenty ten you know, he bonded with his co founder Shane Leg at a safety lecture in which they discussed potential for AI to attack humanity by the year twenty thirty turns out to be perhaps a prescient projection. At least the capability is going to be there, whether the AI attaxs is a different question. But I mean anyway, the point is Demis Hasabis was thinking about safety since the beginning. And so when he sold his company to Google in twenty fourteen, a condition of the sale was You've got to give me safety and ethics oversight board. I can't let AI be rolled out into the world just on the say so of the Google Corporate board. There has to be these independent people from outside. They mentioned Barack Obama as an example when Barack Obama was leaving the presidency. Could we have somebody of that stature who would be on the board saying when it's safe to roll it out That was Demis's vision and it was coupled with another hope which was that all of the major scientists would come together in one single effort to roll AI out into the world, so there would be no competitive pressure to go unsafely and too quickly, right? It turns out and this kind of transpires through the story that I tell that all of Demis' Optimistic stories about how he was going to make AI safe, they all crashed and burned The idea of just one lab Building AI turned out to be a pipe dream It turns out that humanity is too tribal and competitive and dispreutitious. there will be multiple labs when you are kind of confronted with the prospect of being able to build a god machine plenty of different sects of worshippers trying to do that, right and the idea of oversight within Google Ultimately the Google Bard would not agree to giving some outside Grandes a veto over how they used this technology that they were spending billions of dollars on developing They weren't going to do that on a fiduciary basis obligation to their shoulders, they couldn't. They felt Right? So the point being that you know, The experiment that Demis ran at Deepmind and I discovered all these internal documents which back and forth between the red lines from one team of lawyers to the other team about the exact safety mechanisms that they might use and all this of secret strategizing that Demist did to threaten to spin out of Google if he didn't get the safety oversight he wanted. And then the Google Deepmind General Councsel threatened me and said I wasn't allowed to publish any of this and I said, the heck with you I'm publishing it anyay. So it's all quite dramatic. But the bottom line of the story is You know It turns out to be impossible to impose safety restraints Within one AI lab when that lab is in a competitive posture with respect to others. And we saw the same thing play out of course at open AI, but more in public when the safety board temporarily filed Sam Oakman for like five days. So so you know, what this shows us is that if you want to stop a race which has multiple players. You need the government to enforce restraint on all of the players at once And if there are players in China, You need the Chinese government to buy in and also agree to put restraints on their guys when the US. puts restraints on labs within the US France, Canada, that's fine. Basically the US can compel. compliance in those places because Cere in Canada or Mr and France depend on American technology and the American market to function U but With China, you can't compel them So there needs to be two countries, two governments Do a deal. where everybody agrees to put some caution and like checking of models before they're released was the policy of the U.S. government they were going to do none of that And they said I mean, they they even issued an executive order Um banning states from trying to regulate AI in their own way. But then it seems like they've kind of turned on this Last month, Trump signed a new executive order way basically asks companies to to hand over their models to the government, let the government check them and then kind of greenlight them. I mean On the one hand it's progress in the direction that you think is it's the right direction, but also It's not very harsh or strict or strong. It's basically just like, hey, could you please send your model over? We'd appreciate that What do you make of Trump's AI policy at this point in terms of government oversight over these AI models and their safety Given my perspective that government needs to get involved, I've been very much cheered up by what's happened since April when Mythos first came on the scene and galvanized the US goovernment into paying attention and restricting the release. although you could argue, you correctly that on paper the executive order It's kind of a voluntary collaboration system with frontier labs, blah blah blah blah. The reality is it's not voluntary in the least, right? I mean, commerce recently called up Sam Altman at openp AI and ordered him to seek government permission before he gave his latest model to any customer The government has to sign off on each customer, customer by customer This is extremely heavy handed, right? So So I think they're in it for real. The government, they have realized that they Ct let this stuff disseminate around the world without being controlled by government. And so we're going to get pretty tough controls. I think the gap in the system is that they're not talking about doing this in coordination with China Because the U.S policy world has two kinds of China expt that you've got the kind of people who are always hawkish on China People who used to be a bit hopeful about collaborating with China, but then Xi Jinping rose to power and seem to kind of frustrate all those hopes of collaboration. And so that group of former ds flipped and became Uberhawks on China. So you've basically got the traditional hawks and the new hawks, but they're both hawkish and nobody wants to say they want to talk to China This is the problem This is the huge gap in the posture because the US government has done a one hundred eighty on domestic regulation of domestic models and I welcome that The next thing that's going to come Because it's necessary and they're not going to have a choice Is they're going to have to get over their inhibition about talking to China So is that? sort of the solution is getet in a room with Xi Jinping and become nose in tackling this together. I mean, it sounds like kind of simplistic but maybe that actually is the way to do it. The alternative would be you know force their hand in some way, create some sort of policy where you say, no, you're going to get any chips, or you're not going to get this, you're not going to get that Your view is We just need to talk with them and have more of a relationship. It's a bit more complicated than that. I mean, I think you can talk and also pressure on them at the same time. I mean, going back to that Cold War analogy There was a vicious competition between the Soviet Union and the United States at the same time as there was collaboration over proliferation. And so I think there will be competition. And by the way, you know, there are ideas around strengthening the chip export controls and I'm not against that. There is one theory of the case. know economists sometimes talk about corner solutions You can either have a fully pegged currency or a fully floating one. But if you go for some mushly middle ground where it's kind of semi pegged, then hedge fund speculators are going to see that you're not really determined to defend that and they're going to eat you for lunch, breakfast, and dinner It's the same thing with this AI policy. There are corner solutions You could either like, the export controls will be willing to give them up and go talk to them and say, okay, we know you didn't like that as a show of Our sincerity in wanting to work with you, you know, we're going to offer to loosen those controls But in return We want you to collaborate on fixing this non proroliferation risk, right? That would be one corner solution. Or the other corner solution is you don't say that. To the contrary, you tighten up the chip export controls. There's this massive loophole right now whereby If you're a Chinese model buildilder, get this You can train on NVvidia chips, the most advanced versions all day long because the cloud compute that you access is in Malaysia or some other offshore place, which is fully allowed to import Nvidia Chips the most recent sort. This is a crazy loophole. You're telling the Chinese they can't use Nandidia chips But then you're letting them just like Use a data center kind of across the border. It's nuts that that loophole exists, right? So The other corner solution is get serious about the policies that you've enunciated Cut off the loophole and cut off the dillination and put China in a position where it's so weak It's kind of begging for collaboration. Now, I'm agnostic. I'm like We need to collaborate. I'm flexible on how we get there. I think there's different theories. Just going back to Trump's changing of his positioning. It used to be we're not going to regulate, we're not going to have any oversight because we believe that If we do that, then it stifles innovation and we want you know markets to do their thing and AI labs to sort of run free uninhabited etcet. Then mythos happens anthropics model that was a real concern for cybersecurity And then Trump changes his tune and issues this executive order , which you believe in this I think iss fairly so that that actually is like stringent. they do take it seriously Why do you think that happened? What was it about Mythos? Was it maybe something to do with China? Like why did they do this thing that ultimately did amount to a one hundred eighty on AI policy. Simply because Mythos was so powerful, it was very threatening. I mean the prospect that You could take this model and find code vulnerabilities in every single Entity on the internet and then hack it That's curtains for the financial system So that's why they took it seriously. Yeah, fair enough. I mean, I think throughout this topic, throughout this topic The logic of the technology It's going to force governments to do things, which six months earlier they said they would never ever do And that's happened with domestic regulation already in the US. I believe it's going to happen with international collaboration. I've already started to see pushback from I mean, Silicon Valley spent a long time not being friends with Washington. And then in the last couple of years, they became very close friends with people in Washington Um And I would assume that this is going to be I don't know, this is going to cause a rift again because a lot of the technologists said that what we want is government to have no involvement in this in AI Artificial intelligence capabilities and Trump said, sounds good. I'm with you I marries not Um I'm not really sure what that means for the relationship between Silicon Valley and Washington, but I assume, I don't know if you I have any insight into this, I assume It's not going to be great? Well, look, I mean, you've got this sort of Putin in the oligarchs sort of story. you've got, you know endless examples of authoritarian governments u with, you know, big business Titans. And you, where does the power lie and how stable is that relationship? And the answer is it tends not to be stable, point one and point to the government wins because they have the monopoly on coercion And so I think Silicon Valley you know is figuring that out and they realize that the government is too powerful to ignore. I mean, you know Darianaay tried to say to the goovernment, you shouldn't use these tools for certain things like mass domestic surveillance. the government said, get lost. You are going to call you a supply chain risk. And we're not going to be dictated to you. I mean, who won that fight? Clearly the government for won It has been fascinating watching Trump used the full power of that coercion even this week when he decided to step into the proceedings of the World Cup and he got exactly what he wanted and the US abbsolutely got their player back U Just as we saed to wrap up here You have studied a lot of the characters in AI. You wrote your book about Demis Sisavis, founder of Google Deepmind, kind of the open AI before open AI You studied a lot of these characters. And F your research, from writing that book, what did you learn about the people in AI And what has that kind of told you about what might ultimately happen next and who might ultimately win the AI race You've got Sammaltman, who is essentially a commercial opportunist who wants to win commercially or at least survive commercially And you know, his drive is to be a big shot. And he thought of running for Gvernor of California at one point and being a political big shot, but then he decided that building AI was like a bigger big shot And he wants to just, like Put put his imprint on it He's not obviously a PhD scientist, He doesn't have even a first degree because he dropped out of Stanford to do other stuff. It's not to say he isn't anything other than Massively smart. U, but he isn't a deep scientist. Then you've got people like Diama D and Demisisabis who are PhD scientists, who come at this from that perspective, who want to use AI To advance deep science, that's their deepest motivation And I believe it's very deep with both of them. And I believe that's the reason why they are number one and number two in this race. It's good for recruiting the best scientists. It's also good for holdting together and leading a fundamentally scientific enterprise like building artificial general intelligence. And the point where where this came home to me is, you know, when I was talking to Demis one day about his motivation for building AI And he started to say Listen when I'm reading scientific papers Two o'clock in the morning, Sebastian I see reality staring at me in the face Cing at me saying I'm here to be discovered And if I had artificial general intelligence, I could discover the fundamental rules that explain the fabric of reality It would be like understanding all of nature. presumably may have been created by some kind of divine intelligence And so in this sense My quest for AGI kind of like my way of getting closer to what I might call Sebastian Mally is the Paul A. Volka, senior Fellllow for International Economics, the Council on Foreign Relations, a two time Pulitz surprze finalist. He is the author of six books including M Money than God and the Power Lw, which have become investment classics. His latest book is The Infinity Machine Demis Sawa Steepmine and Quest for suuperintelligence. He also co hosts a weekly CFR podcast, the spillover, which examines the ripple effects of global events across policy, geopolitics, economics, technology, and financial markets, Sebastian Thank you so much for your time. Thank you so much. Nice to talk to you. This episode was produced by Claire Miller and Allison Weiss and engineered by Benjamin Spencer. O video editor is Jorge Carti, our research team is Dan Salan, Krin ODonghue and Mia Silverio. Jake McPerson is our social producer, Drew Burrows is our technical director, and Katherine Dilllan is our executive producer Thank you for listening to Prof G Markets from Prof G Media. If you liked what you heard, give us a follow. and join us for a fresh take on Markets on Monday