Government Equity Stakes in AI Giants

Hello and welcome to the Last Week in a podcast where we can near Chat about what's going on with AI? As usual, in this episode, we will summarize and discuss some of last week's most interesting AI news I am one to be a regular host Andre Karenkov. I studied AI in grad school and now work at the AI startup Astrokade And I'm your other co host, Jeremy Harris coming at you from my parents' house actually. You might recognize this background from earlier on in the show. When I used to come here to do it every once a week I'd go to my parents place., happened to be that day. I'm doing it again. So there you go And yeah, I hope everybody's doing good. GlatsonyI national security stuff, AI infrastructure stuff all those things. We're going to try to speedrun this one a little bit. It's going to be an attempt at an hour and a half long episode We'll see how, well, we can stick to it. There is a lot to cover. It's been an eventful week We are recording this on june twelfth. so on Friday and I guess it's good timing in the sense that There have been quite a lot of things that happened and we had time to digest it. preview of what we'll be talking about. Of course, the big story of the week is Claude Fable five being released There are some other deccently big stories like CiriAI Then in business, we've got the IPO train just keeping on rolling with open the eye andthropic And some other fairly big news. We've got quite like a few interesting open source stories we'll be covering. In policy and safety, again, there is a decent amount of discussion going on among the big labs that we'll be talking about So well have to keep research and our things light because there is a lot to get through. Somemer routines live or die by how easy they are. and honestly, if something takes too much effort, I'm out. That's why Groons is my go to. It's my one daily pack of gummies covering my greens, vitamins and minerals. Plus it has six grams of prebiotic fiber, which is more than two cups of broccoli, no mixing powders, no giant pills, no hassle. I just rip open the pack and I'm done. They taste good and they make it easy to stay on top of my health E when life gets busy. Save up to fifty two percent with the code podcast at gros. co. That's code podcast at grunNs. co. This episode is brought to you by Outshift, Cisco's Inubation Engine Today's AI agents operate in silos, limiting their true potential We've been focused on building bigger, smarter models, but scaling up the models is just one approach to improving AI to reach superintelligence together, we need to do more We need to scale out And y actually you have a blueprint from seventy thousand years ago Hans didn't just get smarter individually, the cognitive revolution transformed society because we began sharing knowledge, goals, and innovation. Agents are now at the same inflection point. They can connect, but they can't think together That's why Authirt B by Cisco is building the internet of cognition, performing AI from isolated systems into orchestrated super intelligence By creating an open, interoperable infrastructure, OShift is enabling agents and humans to share intent context and reasoning. Cognitive evvolution for agents is here. So go explore the internet of cognition at outshift d. com That's outshift dot com We'd like to thank Box for being a sponsor If you're a company trying to adopt AI, you're likely facing a common challenge OCITols are great at public knowledge, but they don't actually know your business, your product roadmaps, your sales materials, your policies, all that stuff And that's where Bx comes in Box is building the intelligent content management platform for the AI era sererving as a secure essential context layer for Bxess AI agents to access the unique institutional knowledge that makes a company run The Power of AI doesn't come from the model alone, it really comes from giving AI access to right enterprised content. And that's where box is perfect. It goes beyond file storage. It connects content to people, apps and AI agents so teams can turn information into action. With tools like box agent, box extract, box hubs, and more, organization can accelerate knowledge work, pool intelligence from unstructured content and automate workflows. So if you're thinking seriously about AI, think beyond the model Your business lives in your content, Bx helps you bring that content securely into VI era leararn more at box d. com slash Ii. So let's go ahead and get to it with tools and apps. First up, we've got Cad Fable five and Clad Bifos five. So Claude Fable five is the public you know, open to anyone version of Mythos. Myhos, we've covered for a while now, right? It was announced I sureort showed off by andthropic I think now a couple months ago. not initially released to a wide availability because at least Anthropic made the case that it was too dangerous because of its cyber capabilities So they held it back and Fable five is meant to be the sort of safe to release version that has some safeguards in place to not allow it to be misused for things like cyber attacks or biological research thingsings like that And on the benchmarks, it destroys basically. It's like crazy leap, but we haven't seen it in quite a while. So just give a few numbers. On aenta coding, you get from sixty nine percent to eighty percent moving from cloud opus for eight to Fable five slash slash myifos five on Aent accoding frontier code, you get from thirteen percent to twenty nine percent And you can keep going down the list and see all meaneaningful like pretty big jumps. And I think more importantly There's been enough discussion now to prettyretty much I think The general consensus is this is like The real deal. like fable five is a big leap. Eone I've seen discuss it in terms of their firsthand experience says that This model is now able to be handling really complex stuff and sort of trust it to deliver. on it in a way that was not the case with prior models Yeah, it's a new level of abstraction unlock, right? That's what's been happening where first it was text or code autoc compleplete wr right? So you're like, okay, now I don't have to write individual functions. Then it was like, know, entire libraries and now it's really like the whole app top to bottom that you can kind of just vibe code and tell the model to Do it better and it kind of just does. We've definitely seen that internally. likeike again, you're adding that extra layer of abstraction, redefining the job of the software engineer, the job of the machine learning engineer too, which is part of this, right? Those are the benchmarks as well that are being shattered as well. a lot of these internal AI R and D benchmarks. Speaking of which, so this brings us to this sort of like system card picture where we talk about the risk picture as well. The capabilities are really impressive As Andre said, you know, the vibe check checks out. It does live up to the really impressive benchmarks. But when you look at the flip side of that, so autonomy, right? This idea of roughly speaking, this includes two different threat models, right? So it's not just like loss of control over AI systems. So in other words, like misaligned AI in high stakes, high access settings that leads to catastrophe though that, by the way, they do judge as being an applicable threat model now. In other words, they think it's relevant. They I think that there is enough evidence here that misaligned AI in high stakes, high access settings leading to catastrophe is a thing that is actually on the table with this model. So that's new What's still not on the table is what they call AT two, which is their autonomy threat model number two, that is automated R and D that dramatically accelerates AI progress. That's your sort of singularity, recursive self improvement process. In this case, they ruled it out on two grounds. The first was they don't see they claim an AI attributable two fold acceleration in their own research pace, which is the bar for them Technology, of course, that's subjective thing, right? They do polls of researchers. There's no real benchmark that fully captures this. Already they're saying internally, know they're shipping eight times like eight times more code thanks to this model, which means and the vast majority of their code, if not basically all of it, is AI generated now. But the question is, how does that map on to actual productivity per researcher? They're claiming that's still under two X And until you get there, in particular, until you can substitute for sort of more senior research scientists and engineers, they're sort of ruling that out. and they supported this with a whole bunch of sample day to day uses that show really concrete failure modes. So Th of things like safeguard circumvention, which in fairness, is not a failure of capability, but failure of alignment, which is more concerning, not less, but things like sort of skipped verification steps, reckless actions, failure to incorporate corrections and so on and so for So there are kind of concrete things holding it back from full recursive self improvement, which is kind of notable. They do note a couple interesting things here. heightened awareness of what they call transgression. So they use white box techniques. So These are techniques that allow you to see like crack the model open, you can access the weights, you can access the activations And based on that They find that offtten when the model takes like reckless actions, like dangerous actions in service of the nominal goal of the user The model is fairly frequently aware that those oions that those actions are transgressive, that those actions, in other words are dangerous. So you can actually look inside using all the interpretability tools that we talk about on the show a lot, you know activation engineering techniques, essentially model microscopes and find like, oh man, the model can tell that what it's doing is in some sense undesired, and yet it's doing it nonetheless. So that's quite interesting. Other interesting patterns here, you'll see it stopping tasks early, while internally, if you look at its activations or its chain of thought, attributing it to quotes fatigue or token limits without saying so So you're kind of like seeing these weird behaviors, you know, like there are cases where it's polite to users while it internally registers them as being abusive, All kinds of stuff like this that's kind of, you know, I guess we've gotten a little used to it, but it's a bit frog and hot water. Like it is kind of wild heightened eval awareness relative to prior models we're seeing here. So again, no surprise here, it almost never verbalizes that it's aware of being evaluated too. So that's kind of kind of interesting. And that eval awareness does seem to be genuinely grounded. So you see the models the model putting off indications that it knows it's being evaluated more frequently in training environments that have exploitable graders, like essentially evaluation schemes that can be hacked. And so it's actually using the fact that it's being evaluated to modify its behavior So there's a whole bunch of stuff like that. They did mention, this is not the first time that we've seen this either in anthropic or openp AI, but they did accidentally end up training on the chain of thought during a small fraction of episodes, right? Like this is for all those mundane, annoying software engineering reasons. The whole field had converged on this idea that you shouldn't train on the chain of thought because if you do, you give the models An incentive to generate a chain of thought that looks innocuous, looks good, but still achieves whatever the model wants, whereas if you don't apply any optimization pressure to it, the model will just write what it needs in order to achieve its goal. And if it's planning something nefarious, it'll just write it out in plain English so you can spot it. And they wanted to preserve that for safety reasons. It turns out it's not so easy, right? We talked a few weeks ago about this idea that there's a whole software engineering stack behind training these big models. and it's not at all obvious that you can hunt down every case where the chain of thought sneaks into that. And this is yet another one of those cases. There's a whole bunch of stuff here as well about model welfare. They claim that it has lower self concern. It's more willing than other recent models to choose helpfulness to the user over welfare for itself, which is a reversal a bit of a prior trend. used to see the opposite trend in increasingly capable models And and they've got a whole bunch of interesting things here around like the model snapshots as they train it during the pre training process you're putting in more compute, the model's getting smarter and smarter, right Now what tends to happen is so every couple of training increments they'll take a checkpoint, snapshot of the model during training and then they will ask it questions about its welfare, how it feels for that intermediate model. And it turns out that the base model responses s earlier on would sometimes refer to prospect of having their own values modified implicitly through supervised fine tuning or reinforcement learning that would happen later deeply unsettling, that was the term, and that it quotes fills me with dread. So No idea what that means, but you know, probably worth flagging. Last thing I'll mention is it is important to talk about the chem bioite radiological and nuclear risk stuff. CBurn, right CBRN. So here, they are saying that Mythos five has they have two different categories here as well. CB one This is This covers helping people with basic technical backgrounds, they can deploy non novel non novel weapons. And in this case, they say that Mythos five does have CB one capabilities, and though they judge catastrophic risk, which is like that's a high bar as low but still non negligible. So that's comforting. They do find that they're able to materially improve the performance of world class human experts. They say outright, that world class human experts substitution, substitution may now be possible in a few areas related to the development of novel bio weapons novel bioeapons. That's pretty amazing, like not in a good way, but you know, they're coming out and saying it. I think lots more to be said on the bio side. That's a lot of what I've been working on in the last couple weeks. But man, I think like cyber is really bad. But when you wait for the bio stuff to hit, you can't push out a software update to cover patches on bio. likeike the stuff is s going to get wild Yeah. so a lot to say their system card is like freaking two hundred pages hundred pages. There's a lot that they released about this model in terms of its capabilities and sort of empirical observations. but Of course we cannot get into all of it. I do want to take a second to give my vibe check on the model which I've got to say I'm a little disappointed actually. What I found with both Ous and now Fable is they're very capable at execution of various thingsings and Fable is the more reliable one now at various thingsings, including working at Ascade, we write c. have The model is write code to make games and now you can one shot V impressive things as people on Twitter have been posting But when you get to like deep ideation and trying to get like insights and like leaps of thinking on problems like for instance, optimizing a recommendation system via some novel ideas of how to use the data or but experiments to run or what data to look into. it's like Out of a box, the model is not very creative. I have to like provide all the creativity myself. Now maybe if you do some harness engineering and you do bps optimization and I assume Tropic has a whole effort to make theseel models be good at research by you know, figuring out how to get them to be creative and so on. And we've seen Th complicated systems have like hypothesis running and like Yeah having many, many prompts, many agents all collaborating to get novel results, which in practice usually were not that novel. So I think there is a bit of a gap in the benchmark or I don't know what you call it landscape to understand kind of this slightly nuanced thing of It's very, very capable, but at the same time, veryer disappointing when I actually wanted to do the high level intellectual work in a novel creative way. So that's my liap shack No, I mean so I do agree. I think it does push the level of abstraction up in a meaningful way where you know, you're now like just second guessing the execution less. Like there's less and actually like at least what I found is fewer cases of the sort of trying to like get away with not doing the work that sometimes four point eight would do, which is it's kind of nice and it's part of that hands off experience But in terms of there is this kind of thing, so I'm also doing a writing project on the side these days. And one of the things I can't help but notice is like the output that I get is It's like, it's like stubbornly Actually there it's alm the opposite. It's good at generating ideas for great ways to phrase things But It will often kind of get stuck in a rut of its own making and get too flowery. And I feel like that you see that failure mode when it's codating. It's like the same kind of thing. It tries to get too fancy sometimes And I don't know what the fix for that is because like you know, you want some measure, I guess, of just the complexity of the code base that you're generating. and also like principled, that's the other thing. prrincipled like software bones in the background. L I find myself having to coach it through You know, like you should adhere to like this principled structure. Don't just like go and build the thing. I'm not yet at the point where I feel comfortable saying it, Hey, just go out and build the app. I know a lot of people are saying that. It's just that I find when I do that, I end up regretting, like it doesn't end up being scalable. So I end up finding myself having to go back and be like, no, no, like the bones are wrong. Like let's spend some quality time together to build it up It's a massive accelerant still, but it feels like that's the That's anyway part of what know what the bedrock is. I don't know if that maps ono what you experience but it's similar in nature where basically kind of you can't let the model do the whole thing, right? And that's the the dream for some people of you can just have a model take care of stuff and you don't have to think too much, right? But I guess It's very much like it can do a lot of work for you, but you still need to drive it. for many sort of more complicated tasks. You can build a website, no problem. It's it's can't sort of do it harder intellectual work. A least say it can, but like in one way It like likes one direction and it picks it and it goes in it and then it can't sort of break out like Often I you, se some approach. And then it like gets stuck in that approach and it can't jump out somewhere else. Now maybe there is a way to engineer these models to work in such a way, but until that happens or until you use a harness that makes it happen for many tasks that require that sort of reasoning, these models can't automate it entirely It's like it's like the thing that's missing is almost the open ended learning component that ability to break out of the local optimum and just be like, hey, I'm going to look for novelty now or I'm going to optimize for something a little bit different from just like my opic target fixation, but yeah. Going back to the sort of more consumer side of things, it does cost twice as much as Ous. So ten dollars per million input tokens fifty for a million tokens, which is a lot. I mean these are not cheap models, by any means probably the most expensive But your models We don't know too much about how they are trained or what they include, of course But it is generally agreed upon that this is a new base model Perhaps why they give it the five moniker. And that is notable because there's been an open question of like Are we now kind of out of a post training world or is there still some juice to be had for P training and training new bigger models And Maner seems to be ph andthropic about you still can pretty impressive gains with the pre training, which You know, was a real challenge. We It took a while to get here for the whole industry. Oenir tried with GP four point five At the time, the impression was like RevbeCheck was not super impressed So it looks like Ontropic did do this scale up. And really reduuced the weights with their expertise. And I think My feeling is if you are like XAI or something If you don't have the in house expertise, ye to scale to fable, it's not going to be easy. So it is yeah, definitely a leap and now the question will be can deep mind and open AI keep up. That's the so one of the big shifts recently has been this anthropic seeming to pull away on the quality acis, if not the cost acxis. And know we'll be talking about, I think, a story about openp Iye lowering their cost dramatically or their per token costs, which is you could view as a very bad sign. L you could view it as a very bad sign. It depends on what the source of that is internally If it's, you know, exquisite advances in distillation, wonderful. But like if RSI is the goal, which to be clear, for safety and security reasons, it probably shouldn't be. But if it is going to be the goal, you're going to lose if you're just competing on per token costs, right? I mean, like you need exquisite tokens. That's what this whole exercise has been showing us to make significant advances in like, know kernel optimization and other things that go into AI research. So I think it's a really challenging time to be open AI. notot a coincidence we'll be talking a lot about that IPO here too. There implications for that IPO. their implications for Sam. in particular wanting to preserve optionality on does he pursue an IPO? And maybe does he not? because the scrutiny of the public markets may not be the most welcome thing in the world if in fact, things don't look as good under the hood. Especially given that they're kind of competing to IPO on Tropic and open e on the same time and every single thing on Fropic does is going to affect receptions of open eye and IPO right by its nature is you go on the market and the market decides how much you worth, right? Yeah. So the sentiment, the expectation, matters a whole lot. And these Ias are now priced at such a high level that it's it can be a little bit winner take all, right? If you have a basase best AI out there That's the winner. so Anyway, we'll discuss IPOs a little bit more soon One more thing I'll say about Fable that's worth noting, this release was not without some controversy Those safeguards around biology, chemistry, cybersecurity, and for tier LLM research. were' very severe. So many people in the community commented on how you can kind of basasically not use this for any LLM research of any kind in into sciences, you kindind of can't use it for biology research or chemistry research or even cybersecurity research, it's very, very Guardwils are severe and People were not pleased to see that. And it was not like a very good rollout where initially it was sort of hidden phanthropic had to apologize and say, okay, we'll more visually, more visibly go to claud Opus for eight when things like LLM Research comes up Yeah, this was a bit of a Key releasing It's something a topic we willll have to kind of get good at because these safeguards are gonna Be there now, right? And the controversy, I believe so my understanding was that they were allowing people to know, hey, you know, you've just asked a question that's been flagged for se burn risk, like chem bioreadiological nuclear. For those, it would be like, hey, dude, no, we're downgrading you to four point eight. But specifically for AI research, they were silently downgraded. without giving that indication. I thought that was the distinction. The reason you might want to do that, right? if you're anthropic is you don't want to give off a training signal. to other labs that they can then use to optimize around your safeguards and like fool your system because it knows it's like, oh, okay, so I got downgraded Opus four point eight. Let me just like modify my prompt a little bit to try to sneak through. This suggests that they certainly expect some labs to try to do that. That doesn't necessarily mean, by the way, that they expect openA ee to do that. They may. I'm totally like neutral on that. But certainly they expect Depseak, they expect like Monshot, they expect all the Chinese labs that have been doing distillation attacks. that will be part of this threat model. So all of that very controversial obviously, because you're downgrading without saying. And that's the issue. And I do wonder if distillation, I mean, how do you guard against that dillation isn't you talking about LLMs, it's about like generating a trace of your execution Maybe people are like talk for your reasoning explicitly and step by step. who knows, and that's kind of thing. I think it's what they're trying to do is make it so that you can do distillation, but you can't do it on the exact training distribution that would correlate with recursive self improvement. That at least is going to be the argument that they'll make. Recursive self improvement is super dangerous. We don't want the Chinese to do it. and we also don't want Western frontier labs whose security measures we can't guarantee to do it either. And that's actually a totally coherent argument. And I want to be clear, I actually think it's the right thing to do I'll take that controversial position. I know it's getting a lot of hit online. I think if you just buy into the recursive self improvement threat model and think that it's extremely dangerous. You would take that view. So it's just a question about how seriously people take that risk. I think that frankly, there's been a lot of surprise about it where to me, it seems like it's where everything's been pointing for a really long time. So I feel like a lot of surprise is a little bit manufactured in a sense, not entirely, but a little bit manufactured because it's not like they've been hiding the ball on this one. tootally consistent with everything they've been putting out so far Well, that's a lot unfavable and more to be said, but we have to keep going Next up, we've got Siri AI. So Apple had their big event WWDC and they announced Siri AI, which they frame as an entirely new version of theory that is more conversational and capable and the vibe check on this was like, o wow, Ci works. like we got Smart Si finally, which is a big deal for Apple, right? So this is built on new Apple foundation models developed in collaboration with Google and it can read on screen content, interact ws, manage calendars, suggest action from camera. images and handle writing tasks They released a standalone app for series similar to, for instance, the Gemini app. It's also of course deeply integrated into the iPhone ecosystem. There is one slightly interesting thing where shortcuts now support creating complex automations from natural language prompts, which we've seen the sort of like vibe cody thing also being done by Google So nothing like crazy here, but it appears that after quite a few delays, Apple has managed to bu AI, built, you know modern AI into Vi experience in a successful way You can think of this as an embarrassing piece of context, you gestured at it, right? You said like in partnership with Google. The nature of the partnership here is really important. So this is Siri running on a custom version of Google Gemini. And it's reportedly for about a billion dollars a year that they're doing that, which is actually not that much. like when you think about the CapEx associated with some of these builds, like that's a very reasonable amount you know, billion dollars a year for essentially Google Gemini. And so everything else is downstream from that, right? Like Ler answers agentic password fixing, like all that stuff. sure. but the fundamental thing is this is not an in house Apple model. I mean, some of it may be fine tuned probably is, certainly on Apple data, but is it fine tuned by Apple even? Like, is that even a thing? Or is it fine tuned with Apple data on Google Machines or like how does that exactly work? One of the key things here is that Apple is as safe as the advantage that comes from owning the device. That's Apple, right? They have the phone, they have the laptop. That's what they do It turns out that they can actually succeed at making better AI applications than Google for their customers here then maybe they're okay. Like that's the big question. Is the app store going to look competitive with like the apps that are available on Android If so, then okay for now at least. But if not, that's a really fundamental issue that we're now veering into because that whole moe that the app store has, right, where people want to put apps on it in the first place, that gets eroded once you get apps on tap, just generated by AI. So the only qualifying quality differentiator is the only differentiator rather, is the quality of the AI generated apps And that's the big question. So we'll have to see, like it's really unclear, but Gemini is going to set a kind of ceiling on Apple's AI experience. Like Apple can't iterate in the same way they would be able to if it was a fully internal thing. So Yeah. I mean, you've got this whole thing. Obviously, Apple insists that privacy in AI is non negotiable. That's just their thing But the actual intelligence is a competitor's model. what actually, you know, what the details look like really matters here, but, you know We didn't build the brain is a really big omission in the space Yeah. I continue to kind of maintain that Apple did very sensible thing for this whole thing in a sense in that way didn't try to build a frontier lab with Vin Apple which for instance Matter has tried to do, right? And T mixed results so far. Yeah has a decade had a very advanced AI lab focused on many, many research tasks for deep learning. and was very successful at deepplineing research and deploying deeppl learning models for its products. So it's true that Apple is in some sense a disadvantage or a risky position in being reliant upon Google, but At the same time from a business perspective, I think it was the right move not to try to be a frontelab because Apple would have failed Anyway I completely ag. I think there's so there's a difference between Apples is in a really dicey position right now and Apple made a mistake by not building an internal frontter capability. And I agree with you on both counts. I think they're in a dicy spot, but I think it's pretty much the best given the situation that they've put themselves in over a long period of time This isn't something you fix immediately. They did flirt with the idea of having an in house S pseudo frrontier AI shop where their big differentiator was again, how open source, how over the top open source they were, just sharing everything they could, almost as a kind of like learning in public vibe when you looked at some of the papers, We covered them here before, like it had that energy, but it seems like they've kind of deemphasized that and never fully rotated into it I think the main question if your're Apple is like, do you try to compete on the cloud level because that's sort of the closest thing to the hardware ecosystem, some of the supply chains you already have in touch, you know, I think they've been just too PC pilled for too long. and a lot of their data centers actually look like that. like CPU heavy workloads and stuff like that. So that might have been the way to move. Unfortunately, Google has been so far ahead of them, literally generations and generations focused on AI with TPUs and so on. I think gven that they're in the best spot they could be, try to find a way to coexist with this stuff. and add value to it I think that's the best play to your point And This is coming after we've had to settle a two hundred fifty million class action lawsuit over misleading consumers about Apple intelligence availability and performance. They announced Apple intntelligence two years ago at twenty twenty four WWC Basically, it's only now that they've managed to pull it off Now if they did like pull it off here from what I've been reading in a pretty notable way, like Siri works and works well, which cannot be said of early bred or early, you know, Google L LM releases. So maybe taking their time paid off here, but Clearly, you know, it took them their time and I guess we're lucky that if anything having Gemini It wasn't like a key thing for Android. L it wasn't that big a competitive advantage. aren't swishing for like from what I know, the market share hasn't shifted much just because Ciri wasn't that great Moving on, we've got a story from Google. They have launched Gemini three point five live Tanslate rolling out to Google Meet and Translate. So per name, it is live translator. It supports seventy plus languages. It produces continuous natural sounding Censated speech And it's not a turn by turn system. So you talk to it and it translates in real time as you go. This to me is quite notable because this has seemed like one of a like very clear uses of AI where At some point you're not going to have to learn languages because you can just have an earpiece or whatever, translate whatever is around you We are nearing that point with this release is what it looks like is pushing the same direction as that thinking machines announcement that we talked about a couple of weeks ago that I think is an important direction and it has tons of implications for the hardware stack serving these models, right? I mean this is all about latency. Like when you go into real time, it's like, you need a way to do two things really well. You need a way to really quickly generate an output and you need a way to route more complex challenges to kind of a deeper thinking model. Now this doesn't necessarily have that problem because it's translation only. so you're not having to go and execute on the instructions that are being given to you, which makes it an easier use case and potentially a really good entry point into the kind of thinking machines sort of more complex task handling over verbal channels that thinking machines is doing. I wonder if this is maybe eventually a wedge for Google into that space that we see the same teams or the same infrastructure at least start to get used to push if thinking machines is successful, if that thesis works out into a kind of competitive offering that might be an interesting wayed. So we'll see. but Interesting launch And we've got actually another story from Google. The title is Google Just fired a warning shot in the AI subscription price wars. I don't know if we price was but maybe the very beginning here. So Google cut its Google AI plus subscription price from seven dollars ninety nine cents to four doars ninety nine cents for a month while doubling the storage includes from two hundred gigabytes to four hundred. So this plan includes Stuff like video generation via omni Flash, it includes notebook alam and of course includes talking to Gemini slightly more than by default, So it's an affordable option that gives you a decent amount. It doesn't give you like using Gevenite Pro for coding and so on. you're not going to be Token maxing had this guy, but It's a veryy respectable offering and it as the title says, I guess, it's interesting to see if The price war is going to keep heating up with these kinds of things. Because it's Google and because they're offering you storage as well, I think one of the challenges that you run into is Google doesn't necessarily have to be quite the best in order to give, say, open AI a run for their money at the same price point if I'm getting storage from them and a wholech you know, Google one, a whole bunch of other things That's a bit of an issue unless opening ee can move into that space too. and that's a really big space to move into So yeah, I mean, a lot of that like, you know, It remains to seen. I'm sure Sam has some tricks up his sleeve. structurally, This doesn't necessarily look like the best position for him to be in in a lot of respects. And Thropig doesn't come up so much in this conversation because I don't know anybody who today is doing complex agentic stuff who is trading off Gemini against Claud. So it's just a different category in a way that Codex and Claud are not. Claudd code competes with Codex. And so I kind of see like if openen A is trying this pivot towards, will'll charge you less And that's our AI for all thing Okay, but now you're going to be competing in a really vicious marketplace with companies like Google who make infrastructure their life an obsession. So Good luck, but I like I think that's a that's a challenging spot to be in And speaking of that, moving on to applications and business, we've got open AI confonidentially filing for IPO on the heels of SpaceX and on Propics. So a topropic filed on june first This is now eleven days ago. Open the Ee has filed earlier this week With these filings, we don't get a ton of information because it is Have their ch all But I believe you'll presumably see more closer to the time of IPO. The timeline of the IPO itself is not cleue yet. I think they've indicated sometime this year as the target We are expecting to see VIPO gettingetting up to like near trillion as the initial valuation per their private fundraising valuation It's now a race. SpaceX, we know, will be doing their IPO soon, maybe even within weeks on Fplicon opPI, still unclear and lookooks like Antthropic might be targeting as early as October. I don't really know the business details and the complications of what has evolved here, but As we've been saying all this time, like Oly Anthropic and this IPO race is very much a real thing. And opening eye seems to be in a bit of a tough spot W now Yeah, it's also it's kind of interesting, right? So what opening eye has done is they filed a confidential S one. all right. So confidential S one is something that the way they've set it up is it gives them the optionality to IPO without requiring to do it. So they're buying the option to IPO And then also doing it in a way that should be confidential And yet, opening eyes is also publicly announcing it. They're saying we expect it to leak. so we're just getting ahead of it. basically. Well sorry, they don't say getting ahead of it. That's what they're doing, but they say we're just announcing it basically And then at the same time, you you' got Sam making philosophical statements and waxing poetic about AGI and economic growth whichich feels a lot like a kind of narrative management play. like you're going to get scrutiny if you actually do IPO. There's a whole bunch of, and there's an expectation that they'll do it. And so if Sam doesn't want that scrutiny, if it doesn't look like you'll be good because he thinks he, for example, might want to raise again in the private markets before going public and might want to be able to kind of spin his own narrative that context with less scrutiny, then that starts to look appealing. So this is it's all reversible, like this confidential filing, and that seems to be part of the hedge. The other piece he introduced here was something about, I saw him say something on Twitter about depending on how close we are to recursive self improvement, it may not be desirable for us to be a public company. You know, the implication there being about control, recursive self improvement is this very dangerous thing We want to have unambiguous control over this and private company can do that manage that better is kind of the subtext you know, sure Also that seems to awkwardly fit with some of the other things we've been talking about here where, you know, maybe the scrutiny isn't the best thing in the world. for open eye right now. it's hard to know. The fundamental reality is these folks are all fighting, as you said, over the same pool of capital. Bankers have been making that very clear, apparently, to open AI anthropic. There's an early mover advantage here. The first to list is going to set the terms for how investors think about the AI sector and get first access to huge pools of capital. The moment one of these companies goes public Anybody who's thinking, you know, I've been cut out of the frontier AI game, finally I get to go in, they put their money on frontier AI, which is the category really that open AI Anthropic and SpaceX and, who gives a shit, they all represent it. That's kind of how a lot of retail investors are going to be thinking about this and some institutional as well. So that's kind of the risk, you know, they shoot their wad, so to speak, of cash at one of these companies as soon as it comes out and then they think of their job as being done you know, we're in we're long frontier eye and we're good to go. No need to, you know, keep dry powder for the next company Next up We've got Jeff Beetos Prometheus raises twelve billion do to build an artificial general engineer for the physical world. So this is a physical AI startup co founded by Jeff Bezos and Vig Bajj They have raised this twelve billion at a forty one billion dollar valuation with funding from Bezos himself, JP Morggan Chase, Goldman Sachs and Black Rck. of banking institutions not re R Interesting. This is after an initial raise of six point two billion when they launched late last year so prettyty big raise pretty soon after the announcement. And at the time, it now has eighteen billion dollars Artificial general engineer is kind of interesting. They are saying this is software aimed at automating the design and manufacturing of complex Physical systems such as jet engines and drug compounds So they're not trying to compete onve alamp frront. presuming we need to do some of the fundamental research necessary to apply advanced AI to these things, but if anything, we are competing with deep mind and there' spinoffs that deal with physical sciences and not so much philanthropic or open AI it's so there's something This is the wrong, I don't mean the way it'll sound, but there's something old and slow about what Jeff Bezos is doing here, right? It's the Peter Thial thesis like Aams over bits. He's basically surfacing that like, look, there's vicious competition at the level of you know automated software, you know, chat bots, coding assistants, general models, that stuff I'm going to focus where I'm best at on what I'm best at. So're if you're the CEO or former CEO of Amazon, you know, you want to look for things that involve building stuff, you know, capital intensity, regulatory gatekeeping. That's a sweet spot, right? You know how to deal with sock two compliance and like the Fed ramp and all these awful things you have to do. proprietary physical data is another big one, Data that you can only get by touching the real world that's the gold. and just anything that involves like just difficulty of operating with atoms, right over bits. And so this could be part of why we're seeing the old and slow institutions instead of VCs, the banks piling into this one. You can in some sense kind of derisk this a little bit more because you can look at like, okay, well, you know, this is an unsolved problem and it involves stuff that Jeff Bezos, of all people will understand deeply. It involves moving stuff around And that's good. we can price that out. We're used to pricing out the construction of buildings, the development of supply chains, the kind of engineering, manufacturing, all that stuff. And so that may be part of why this is happening at the same time I have to imagine you'd want if you're Jeff Bezos to court the seequoias, the Andrason Horrw with.ike I don't know why they're at least on the cap table. The article doesn't say they that they're not, but they're not listed and that's weird when you're listing JP Morgan, Goldman and Blackroock. So there you go. Anyway, we'll see. Another thing to flag too is this is a really big raise as a percentage of the cost of the company. They're giving away twenty five percentage of the company which You might go, well, wait, doesn't anthropic like they'll like give away less than ten percent? L their latest fundraise was sixty five billionars on nine hundred and sixty five.ike you know that's nothing. Well, yeah, that's true, but hardware is hard, as Silicon Valley likes to say, hardware companies tend to have to raise more for their valuation than software companies. It's just because it takes longer to build stuff. you know, like that's the cost of doing business. So that's kind of what's going on here and also why it's so strategically relevant that Jeff Bezos can afford to put in his own money to bootstrap things. So There you have it And speaking of raises, we've also got some news about Deepsek. They are seemingly set to raise around seven billion dollars in their first external funding round. The post investment valuation is expected to be between fifty twoars to fifty nine billionars dollars which is actually not that much. It sounds to me like for a while now DepPak has not been seeking outside capital. They have been funded by sort of their parent hedge fund high flyer. So this, I would imagine indicates that they are looking to scale, looking to, you know goo bigger than you can go with just one hedge fund providing you with capital Yeah, apparently this is another one where so Lan Won Feng, who's the CEO of Deep Sek, he's put in a bunch of his own money, twenty billion yuan, which I think is dollars Sorry. That was a terrible joke. Anyway, So he's putting a bunch of his own money. And also so there was an announcement I saw like a recruitment announcement that was circulating on Twitter that was gesturing at Deepseeak saying that they're looking to build clusters now on the order of a gigawatt scale That's catching up now to the Western frontier. It's not quite at the same same rate. And important to note that in fairness, Chinese chips don't convert energy into flops as efficiently into compute as efficiently as Western ones do because we get to use TSMC's fancy nodes But still, in terms of orders of magnitude, like you're kind of getting there. So China has this big energy advantage that's part of what Deep Seak is going to be able to tap into The investment lineup here is basically a vertical integration play. You've got tenencent, right? You've got JD to bring basically so jD dot com brings distribution and cloud. They've got massive infrastructure for that. You've got the national AI fund in China that brings state backing. And then there's this company called CATL that I hadn't heard of before. Apparently it's a battery and power equipment giant in China which is the kind of odd one out seems to gesture at what the strategy might be. They've been pulled into data centers for AI, and you obviously got huge opportunities there for energy storage solutions for AI, workloads, stuff like that. So really like you're looking at every layer of the stack participating in this fundraise, which Kind of makes deep seek, I mean like Huawe is the only other company that's nearly as vertically integrated. Deep Seak's been relying on their chips historically, that partnership like I wonder where that's going to go and if that's going to be another kind of like Frenemies situation, the same way we've seen NVidia and some of the giants like Openion and Anthropic kind of like like each other because they have to, but only because they have to. You know, that's part of what I think is going on here too. We'll see Speaking of that, the next story is that a Huawei led team has claimed that it post drained Deep Se' one point six trillion paramometeter model using one thousand Asid se chips. So meaneingful part here is that they had a big cluster of Chinese chip. My name is Shanna Mald Donato. I'm the founder of Yaoi, a gift shop on the lens of artists and handmade objects. I chose Shopify because when I was testing other platforms, it was definitely one of the most user friendly. It was important to me to think about where we would be in the future, all of the tools for reading your sales like planning inventory, They're just right there on your dashboard For anyone starting a small business, the biggest thing I can tell you is it doesn't have to be perfect. Shopify can help you build upon it Start your free trial on Shopify. com I started Ornaud in twenty thirteen and we make bike apparel The best part of Shopify for me is our ability to run the business as essentially non technical people. We're able to admin everything on the back endnd, front end, and sell things online easily. If Shopify were a bike accessory, I think it would actually be the bicycle. It's the thing that you'd do the thing on We run the business on Shopify Start your free trial on shhopify. com being used to post trarain a big which Previously, we know Deep Seak has tried and has used these chips to some extent. But it hasn't been, as far as I know proven to be the case, you can basically have them be big driver of major training runs in the same way I think Via could be, you know, even going back to August twenty twenty four late twenty twenty four I wouldn' be surprised if deep seek was a big part of why Hua was able to make with chips work in the first place, they have like We've seen we've discussed their research papers just like going deep, deep, deep into the weeds of hardware and having like explicit bequests from hardware makers So this does indicate that like You know, maybe that ecosystem is getting to a place where you will not require NVidia chips At least at the level that Depsick is at currently Yeah, one thing to note, know for a little while, we've seen Chinese companies, including DeepCa confidently do inference, serve their models on Huawei hardware Huawei chips, Ands but not training. And then that's because training is like fundamentally harder than inference as a task. Inference, you basically just like sit there with your catcher's mitt and you wait for the input to come to you and then you just turn on an output and you got to bet really good at doing that really, really fast, right? That's basically it there's whole bunch of complexity like all things, but that's roughly it. Whereas training, you know, like get the data in, you chop it up, pre process it, you send it out to, you do your RL rollouts, you then score the rollouts, you pull them back in, you know, like you're doing all this insane orchestration and that's a big part of the reason why there's just a bigger mode that Nvidia enjoys and why you still really haven't seen Even with this article, you haven't seen pre training done That's one of key things that sort of like buried in here. So it turns out that this headline number, one point six trillion parameters and we're training it, they say It sounds impressive, but the fact is it's actually post training. So there's a kind of ambiguity here about where pre and post training begin and end, of course. But Deep Sk's documentation puts the pre traraining corpus at more than thirty two trillion tokens. in this case, it seems like they did not do that on Huawei gear And it's more like yeah, like a kind of not fine tune, but like there's,, there's more going on, but still post training. So that is impressive. There's an awful lot of RL and orchestration involved in post training, but it's not the full thing. And also like there's a history of stumbling here, right? So like yeah, you mentioned back in August, there was an issue that Deepsee had with R two where they couldn't train it on send chips with Huawei engineers on site. and then they blamed performance, they blamed gaps in basically the equivalent of CUDA, which was not yet where it needed to be on the chips and so on and so forth. And slow chip to chip interconnects, which for Huawei is a much bigger deal than it would be for NVidia because Huawei depends on meshing together huge amounts of chips to make up for the fact that each chip is suckier than their Western equivalent. So when you have a chip to chip interconnect that's weaker, that's an even bigger deal than it would be in the West. and in the West it would be a big deal. So there's stuff that they must have improved to get here. I'm not poopooing this at all The fact of the matter is it's post training. It's not a full pre trained soup to nuts And speaking of compute, I'm just out of roll with these transitions. Google will pay SpaceX nine hundred twenty million do per month for compute. So this is following And Froplic making a deal with SpaceX as well for I believe similar numbers, like in a billion range. hereere, Google will pay SpaceX nearly a billion dollars per month from october twenty twenty six through June twenty twenty nine for access to approximately one hundred ten thousand NVIidia GPUs Google has described a deal as a short term bridge capacity agreement to meet unneexpected high demand for its Gemini entnerprise agent platform, which It is surprising. I would not have expected Google to have that This is coming, of course D close to space expected IPO and it again positions the AI part of SpaceX as like We are providing compute for the real big boys of frrontier AI which may or may not be a good place to be at, but the GPUs are not going stay useful, right? So when you're in this business, if they want to go in, they can go all in on being a compute provider But we haven't seen indications that strategically, that's where XAI wants to be And to your point, there's another company that's gone through a similar transition. It's less obvious, but you mentioned them. It's Google Google, I'm old enough to remember when TPU's were supposed to be a super secret squirrel project inside of Google and no one else got to touch them, Well, guess who's rolling that out and trying to become the infrastructure layer for the AI revolution, right? So this is SpaceX trying to live up to that thesis in fairness, they put in their S one, where they're like, look, basically we're going to make Dyson spheres, man. We're going to go out there and put data centers in space. They even came out with an early design O basically a kind of early space data center concept. So they're really trying to push this and make it happen. I'm sure it will eventually, the question, of course, is when and will it be price competitive with just slapping a bunch of solar panels on the surface of the Earth. But yeah, so it is, in that sense, pretty interesting that the two companies that are playing this very game are in business together in this way. It's not a small amount of money, right a billion dollars a month, nine, twenty million, okay, that's pretty decent One caveat though So either party can exit this agreement within ninety days notice. and there's a separate earlier escape hatch too. So SpaceX, if they failed to deliver on the GPU capacity that they've committed to by late September, Google can terminate the agreement immediately after a one month grace period, or they can just accept the available GPU's with proportionally reduced monthly You know, this is kind of like the anthropic one, right? It's It's not no one's locking into a long term commitment here and to your point Those GPU's will hum for as long as they are useful And they do age out. Things are complicated there. That actual market cost of a H one hundred has gone up actually in the last a little bit, which is weird. But you know, there are all these questions about what does the actual value of these GPU's look like over time But still You're right, SpaceX is looking more and more like a neoc clloud every day. The cursor play is clearly an attempt to kind of get back into the frontier game. It kind of see when you think about what is the story behind the S one, What is the story behind the IPO? the story is data centers in space. Like there's no other there's no other story there. It's the only way you get to the ten trillion dollar value that people are assuming when they actually invest at one point seven seven trillion. So I think it is really interesting. It's working on Earth. Eon is clearly the best in the game at rolling out data centers quickly when he shouldn't be able to on the planet. Maybe that continues in space, who knows? We'll see Yeah, and that is explicitly the case being made. So on Monday, there was this I believe like thirty minute video posted on X where the new story title is Elon Musk shows off AI data centers SpaceX wants to send into space. So The pitch, as you said, is data centers in space. This is why SpaceX is AI They are the only ones that can build data centers in space and you will need data centers in space. They provided some kind of kind of renders and a few details some size estimates. they're saying that be large sonal panels and liquid radioators SpaceX does have an existing starink starlink program of like a ton, a ton of satellites. I forget how many SpaceX is saying they're planning to launch up to one million of these AI data center satellites and is building a giga set factory in Texas that is Not to be confused with a tariff initiative, which they say eventually will have specialized radiation hardened chips. Although initially with satellites we use NvidiaGPUs because Terraifab It'll probably take a while. I Data center is one thing.. Bab is an entirely different thing So they are positioning themselves explicitly as this orbital AI data center play and I'm actually very curious, Jeremy. you've done a lot more deep diving into hardware stuff broadly speaking I've seen, you know multiple takes on this as like This is just another one of Elon Musk's pipe dreams promising like, you know, the hyperloop or whatever that isn't going to play out. It's sci fi and it sounds cool, but it's It's it's nothing but puff air. and it I' from what I can understand about the physics involved, it doesn't seem viable to do Like maybe you could do inference, maybe Not training, right So what is your take on this? So my take on it is I am tired of betting against Elon and then like, you know, the reality is that like the Tesla shorts got nuked, the SpaceX shorts got Absolutely. on the stock. I'm not betting against ace For sure. I guess where I'm going is Elon's pipe dreams include the Boring compompany. They also include Neurolink, which is having a bit of a moment. They also include openp AI. They also include SpaceX. they also include Tesla motors These are a lot of impossible things, like efficient reusable rockets that take off andpay. Like I think we've been desensitized a little bit now far.. So to be fair, if anyone has a track record to do impossible things that on paper, no one believes will happen You gott to give it to him. Ellon Musk is the guy who has done that the most too any mortal man bound by the actual laws of physics, like I would completely agree with the analysis that you've made. And I think there's a good chance that so the classic Elon problem is that, you know you We were supposed to have self driving cars on the streets everywhere right now. And it was supposed to all be Tesla', by the way, not WayO', not like what? So yeah, you know, there's there's kind of like an issue there, right? And it's a delivery time issue. You get magic, you just get it A couple of years after you're supposed to And I think that the delay scales with the challenge of the technology You know, like I forget when we were supposed to have that Mars base or that first trip to Mars or whatever, but you know, its stuff like that. I sure it was this year, it was Mars, maybe around this time. We're in June, Andre. We're in June, okay? Yeah But yeah, no, exactly, right. So I think that's kind of part of this. I don't feel strongly either way. like I'm looking at his you know, XAI is another thing that you can argue has not gone well But he's still trying and he's pivoting SpaceX in a way that could like it's all I'm again, I'm like confused and tired of betting confidently in any given direction on any given play with Elon But certainly, this is a really hard space. I think the key thing is, this space is also very old. So if you think about like TSMC from a Fab standpoint has dominated forever, that intrinsically is begging for disruption. Like I actually don't care what the details are under the hood. Well, you know this Andre, you're in Silicon Valley. L this is how it works, right? What is Y Cominator? Y Cominator is a factory for startups that think they can do something huge better than the incumbent And the only thing they have going for them is the fact that they're willing to swing for the fences and do crazy things. You're competing with Nintendo right now I think you're probably going to kick their ass over time where companies like Astcade will end up really because they their AI first or like they just breathe that space in a way that the incumbents don't. And it's not the incumbents are trying. it's structurally engineered around something fundamentally different. So if you want to think of a comparison point Tesla motors versus general motors. like there hadn't been a new car company that became a big success in like seventy years. And so you know, in that sense there' echoes of that too. So this rhymes a little bit with what we've seen Elon do in the past in a way where I'm like I don't know. I genuinely don't know I'm not going to be able to pull that out, but I think you know, who knows Yeah, it's maybe the most impossible thing he's promised to deliver just from a physic standpoint. So Maybe he could do it, but which is saying a lot, which is saying a lot Yeah, we'll see, but that's the pitch Ono projects and open source we got just a couple quick stories here. Google has released a new Gema four twelve B model that is designed to run on any laptop with sixteen gigabytes of RAM. So this is Filling in the gap between its mobile optimized model E two B and its larger models twenty six B MOE, I think we covered mobile laptimice was run just recently. So this can be run on a fairly, you know, not huge laptop, sixteen gigabytes of RAM or VRAM is not Huge which is interesting to me. I mean, Google does have laptops, I think. they do compute. And this kind of on device LLM is something that there is a real community around doing. And for certain applications, you might imagine on device LLMs would be very valuable. So so far if it hasn't been something open AI on Tropic anyone has care to do much in the space of, but Google is getting some advancements here and releasing models that you can run you know, in competition with Chinese providers, Chinese models that also are very much on device. This is also there's a weird thing going on architecturally with this model. So like usually when you use a multimodal model, you will have an encoder, right? Like basically a dedicated submodel that is taking the input, you know, text, audio, video, whatever and like pre chewing it so that you end up with essentially like a latent representation that is already kind of compatible with the language the way the language model downstream wants to process it. And what they're doing right now is they're throwing that out. esssentially they have this just like well, it's like a thin thing that they've replaced it with. It's a tiny, it's thirty five million parameter projection really that splits images into patches and projects each one into just the downstream model's hidden dimension with just one matrix multiplication, like one step One small thin step. And well, I mean, so yeah, that'll help with latency. It'll help with memory, but the The reason you use an encoder is because it actually works, right? Like doing that pre chewing of visual features especially means that the LLM downstream does a better job. And if you get rid of it It's kind of a bet that the language model has gotten good enough that it can learn that like visual understanding directly end to end And if that's true Awesome. you know, memory, latency, those improven in potentially pretty big ways But if it doesn't, The way the improvement shows up is kind of exactly where the benchmarks suck at measuring. It's like on real world vision tasks, like dense documents and stuff like that that involve like fine details. And so this is a real vibe check one. Like you know you're going to want to see if you get degraded performance and if you don't, then that's a huge win from a memory standpoint and from a latency standpoint So it's an interesting bet, we'll see where it goes And related, Google has also released diffusion Gemma, slightly bigger model, twenty six billion Mixure of Eress model that is released under Apache two point zero and is diffusion based. So this is a pretty speculative direction in language models where the traditional language model is What utter aggressive is a term. It produces one talker at a time goes left to right the same experience you get on ChGPT. Dusion, traditionally we've seen in images where you kind of start with a bunch of noise and then you den noise and you Basically generate everything all at once left to right top to bottom And that can be done in text. You can generate an entire paragraph all at once. startarting move like noise nonsense and then getting the good words over time. That historically hasn't worked. and it's a bit of a mystery actually why diffusion just doesn't fit for text and can work pretty well for images But with this release, we are showing that you can do pretty well. it's not competitive with the LLM based model of the same size that just is built together with, but It is close on some benchmarks like multilingual Q and A or MMOU pro graduate level knowledge And it is four X faster and that's faster than an already fast model. So this can generate over one thousand tokens per second. That's like eleven X, something like opus, for instance. So blazing, blazing fast And with this level of performance, you know, you're getting to a point that if you just want a Chabba to do basic stuff for you, you might be able to use diffusion. Now Google does note in their blog post that this technology actually doesn't work very well for large scale cloud situations, cloud deployments. It's very good for a single device for inference, but actually doesn't work as well when you're scaling. So I'm not sure what the implications are business wise for Google continuing to invest in diffusion It is quite exciting because this like thousand tals per second is a really fun experience when you try it Yeah, for sure. And this is so the reason that they're seeing this sort of relative nothing burger when it comes to deploying it, say at the data center level is that the thing that diffusion is doing is kind of the same thing as the thing that batches are doing in a data center Diffusion lets you look at like your whole your whole input and output like kind of together coherently at once. you're not just generating one token at a time. And what that means is you have a whole bunch of paralyzable tasks that you can send out far out to your GPUs and keep them working And really, the thing that determines your economics at the data center level is are all my GPU's working as often as they possibly can, My GPU utilization The thing is that the way you solve that problem in the data center for traditional autoreaggressive models is through batching. You're just taking a whole bunch of different user queries at the same time or a whole bunch of different samples at the same time. and you're throwing them through the same GPU's. And so yes, you're going one token at a time, but you're doing that for many different samples at the same time, Whereas what this is doing is is doing it for all your tokens at the same time, but for one sample at a time which is why it's a better use case for you sitting on your own personal laptop without a data center to support you. You're going see that speed up delivered there. You're not going to see it delivered at the data center level because the basically memory bandwidth restriction that applies so heavily like at the single user level where anyway, you're having to like load the whole model in your one machine just doesn't apply at the data center level either. So anyway, that's kind of the reason behind it. And then they've got this like kind of structural play here where they have so they have this bidirectional attention plus renoising thing that we I guess won't get into because the time. But basically the model sees the whole like two hundred fifty six toen canvas basically like here's the two hundred and fifty tokens or so. And what it can do is revise a single token that it's unsure about. and kind of like constrain its optimization in that way in a way that just like anyway, left to right generation just can't handle as well. So things like code infilling are just natural fits for this and some planning tasks as well where knowing the outut of the thing is important to refactoring the tokens before. So anyway, it's an interesting use case but I think this is still kind of like You're not going to see these models be the frontier models until something fundamentally changes that hasn't changed yet And I he it's interesting that they're releasing this like fully, fully open source Apache two point zer no restrictions whatsoever. I guess makes sense given that this isn't going to give you intelligence above anything that you can already have And they also are releasing hackable diffusion here, a fine tuning tutorial bunch of other stuff. So's it is kind of giving this notion of like Hey, AI community, let's make some progress on diffusion because it might be able to be quite cool So I look forward to seeing more progress in LM diffusion because on device LLM, but is super, super fast could be like a game changer, like a quality totally different type of experience once and it's not at the point of working yet, but it It's looking more and more feasible with this release Ono policy and safety. firstirst up, openp AI and Anthropic sign letter to prevent AI developed biological weapons. So this is Open AI Anthropic and DeepMind and Microsoft are among the signataries of a public letter urging Congress to pass laws requiring synthetic DNA and RNA sellers to screen customers and orders to prevent biological weapon develop This was organized by the Institute for Progress and the Foundation for American Innovation talking about really that threat model of biological weapons. And I think we're starting to hear more and more and more about biological weapons Jeremy, you just said that you've been focusing on it more. So what's your take on public letter? is it meaningful in any real way Yeah, every bio weapons expert I talked to is freaked out right now. like there is a pretty significant difference between Bio and cyber when it comes to AI, I kind of alluded to it earlier, but like, you know, on cyber, at least you can patch the vulnerability. You can roll out mythos and you know, you can try to give it to central banks and you know to all the big banks and you can give it to all the emergency services and try to get them to do stuff. The problem with Bio is you can't update the software fast enough to, you know, if somebody develops a new thing. And so That's a really big issue. Think about what would it have looked like for those of you who remember the language of COVID, right? Incubation time, how long the virus sits inside you before it shows symptoms while being transmissible What would it look like to have a virus that has a very long incubation time? so you expose yourself to a lot of people before you even know you're carrying it but has super high lethality So we just haven't seen viruses like that so far. Usually when a virus has really high lethality, it has a short incubation time, very quick to kind of be spotted. It's worse at kind of traveling, a low R knot. So high R knot, long incubation time, high lethality is kind of a trifecta that would be really, really scary. and you can imagine optimizing against Nature's natural processes to push in that direction, also targeting people based on genetic characteristics and stuff like that There's a related thing here that makes it really challenging to fight, which is Unlike Cyber, which all lives in the AI sorry, which all lives in like the world of code and software. problem with bio sequence data is that You can both have an attack at the level of the model, so data poisoning or somehow like, you know, make a sleeper agent or corrupt the model to generate a sequence that looks innocuous but actually contains like a really harmful bioagent when you make it Or what you can do is bet on a downstream intervention. So you know like this sequence is actually innocuous if it's made the way it's supposed to be. I know that Thermmofisher scientific who who's actually going to like you know, make the buffer solution that goes into this thing, I have a guy in Thermmofisher and I can get that buffer solution to have a pH that's a little bit different from what it's supposed to be. and in a way that activates some latent thing. So even if you scan the sequence naively, everything looks fine But then somebody acts on acts on it downstream to make it dangerous So this is where like just scanning the sequences is not enough. You need a like physical model of the industrial process that produces the bioeapon in order or sorry the not bioe the you the bio thing, biomolecule in order to actually defend against this. and this is like the nation state game. you know, they attack at every level of the supply chain and they in practice do have people at every company that they need to have people in, especially some nation states. So yeah, I mean, this is genuinely really, really concerning. There's, you know you can imagine the people who would be concerned about this sort of thing. and it's something where hopefully fortunately, there's an actual incentive here for international cooperation on this because bioeapons tend to kick back, right? If you believe the like COVID gain of function theory, which I mean, I think is very kind of well backed at this point then effectively, that's a bioeapon that leaked, right? And a lot of people got sick and died in China. It was a big economic problem for them. And so you know these things this is the reason why you actually do have bioeapons treaties that tend to be enforced, sameame with nukes, same with Chem, you know, things that backfire are not generally things that people like to like to violate And as a related note, we have Anthropic CEO publishes a Lenthfy article. AI is moving too fast and policies can't give up is the headline. So this is Dari Amadeay posting on his own blog, which he's done several times with Lenthfy thoughtoughtful pieces This one was titled pololicy on VI Eonential and it called on VS government to establish a regularity dulatory body similar to the FAA to conduct mandatory third party testing on all cutting edge AI models. So this would cover four dimensions, cybersecurity, biological weapons, runaway risks and automated development with a government having a right to block release that the models will fail AmedA explicitly says that AI may lead to significantly worse and persistent unemployment. And generally, you know says that this is dangerous stuff and we need to be able to have policy that keeps up This is, you know, we covered I believe last week what the Trump administration has done with AI regulation, which was It was kind of a very sort of oh if you want, you can let us know how dangerous your AI models are. It's voluntary. Don't worry about it Here Amade is very much saying that perhaps we want a more strict hand approach where the government has a very active part to play with requiring third party testing and in fact reiring the safety pass before this happening Alongside the article on topropic did announce approximately few hundred fifty million dollars and new initiatives for their kind of research arm saafety research is part of that AI safety research, AI control, that sort of thing One of the challenges too with this is like internal deployments Like internal deployments, if you believe in loss of control are maybe even more dangerous than external deployments, right? You're giving the model a really big leap in the kind of freedom of movement, if you will and freedom of access that it enjoys. And that's kind of where you might expect it to surface first To do that, there's also this challenge of like How do you ensure that labs don't hide big training runs? In Wall Street, often in the big banks, the big institutions, the hedge funds, you'll have like a floor of regulators who just sit like physically in the same building and have access to the different offices and stuff in house. And that's like a really important function to help surface squirrely stuff. Now fortunately, big training runs are really hard to hide. They have a massive industrial footprint, if you will But still,, as the stakes of just trying something radical rise, you need a way to get really intimate access to the inner workings of these companies With that access also comes opportunities for security to be undermined. So there's this really tough balance, right? You want oversight, but the more govies you let into a building A lot of guveies are compromised. A lot of goveies have been targeted and recruited and so on. And so this is a very complicated situation. His take on macroeconomics and tax policy is essentially a permanent AI underclass. So you know, that's not the first time I don't know he uses those words, but it's not the first time that we've seen that theme come up. I think it's just true like The reality is that historically, economic activity has been a function of labor more than capital, and it's been increasingly shifting towards capital. In other words, you make money by owning stuff. L you own now wads of intelligence in the forms of data centers power and GPU's. and now you don't have to get people to do that work. And so the leverage of workers I am, to be clear, a libertarian dude and I'm sounding an awful lot like Bernie Sanders right now and I'm not blind to that, but that is just like if you believe where this thing goes, there is actually a point where Bernie Sanders Maybe not Bernie Sandard, but like, you know, shit that sounds more bernie than I would be comfortable with here. like it starts to sound pretty reasonable, but' so advanced. Socialism, maybe you want to consider it because like everyone's got to be out of a job. What are you going do? Capitalism breaks down. That's's just right. And this whole idea of like, ye, you know, you'll find new jobs is like, well, here's the deal. You're automating the exact function that allows humans to find new jobs in the first place You're automating the intelligence itself. You like faking get automated by some whatever they use back in the days where kids would pick lice out of their heads, you know, like one of those new factory machines who go, o, crap, this factory machine, it picks lice out of my own hair. I don't have to do it myself. Cool. Now I'm gonna to have a think. I'm gonna to have a little think, and I'm gonna come up with some idea, o Ohh, Steam engine. good. I'm gonna to go work on a steam engine. you go work on a friggin steam engine. You could do that The problem is now, the AI is going to think of the steam engine before you get to. And then even if you do, it's going get better at it faster than you. And so like I don't understand this whole argument where somehow we end up with this paradise where every anyay it's a whole thing. You don't need to hear it from me. I don't know more than anybody here. I'm just saying the thing. So yeah, I think that's a permanent AI underclass. We can put Dario down in that column, though he would not use those words to be very, very clear Right, so this post covers actually a few things. so regulation, macroeconomics, accelerating AI's, positive impact state and civil liberties and securing leadership by democracy. So it's got like a lot of policy related stuff and Yeah, covers regulation, covers economics, covers The fact that democracies should win is on Troic's official position AI maybe will be a dominant military power. and if you're able to secure In most advanced AI, you are likely to be the most powerful nation. So It covers all of these concepts. It doesn't kind of put forward anything particularly new except that relelative to before Dario is, let's say, a little bit more worried having seen Myipos. with the sort of progress it has earlier, the position of topic was, you know, we shouldn't require regulation or have heavy regulation of this sort. and it appears they are reconsidering that position And a related release also from Van Frabeck Institute, not fromario, they have released a post regarding recursive self improvement lengthy post discussing it, covering The title of the post is when AI buildus itself. progress towards recursive self improvement and its implications. So it co we touched on this slightly earlier the evidence from the Vin andropic that AI is getting to a point of improving a yeah and You know, the obvious part is is helping the researchers and engineers write through code. That's pretty clear But they do have many examples of it getting better at open ended tasks, getting better at research I have an example from April of it taking a pretty open ended problem and just rolling of it And they get to a point, I think interestingly where they say it's getting to a point where we may need to consider the option of a global pause in AI development, a coordinated effort to make it possible to maybe make it a reality that a global pause is even possible. Now it's not kind of naive, it does cover the fact that you need everyone to be on board for this to really matter and may be doable, but it's not clear. They're saying that in the coming months they will organize conversations where policymakers, researchers, civil society, and other AI companies can help answer some of the questions this piece raises, especially around recursive self improvement and how to create better options for coination and deliberation Yeah. and I will say you read the post, you're not going to find anything too new, right? We've kind of covered this has been you know let out byanthropic in bits and pieces, all the stats about automation, how much of their software is written by AI, what the uplift is that they're seeing, all this stuff. So in some sense, I mean, you know, I think if you've been following the show, you're kind of If you were that way inclined, you're there already. There is this sort of sort of tying it to concrete timelines that they're doing where they're saying basically, ye, within a year, within two years, like we're kind of going to get there and like we're not sure it's going to work out And so in a sense, I think this is anthropic just sort of like ending kind of ambiguity that arguably had inserted itself. because you they talk a lot about recursive self improvement being the freaking goal Like that's they talk about it being horribly dangerous, which I think is right. And then they talk about it being the goal, which is also inescapable if it's open eye's goal, which it is. if it's, well, Google Deepmind's goal, I'm less clear actually on what the latest is there on. Do you ever to remember if we've heard anything from GDM on RSI? like are they? Don't think they' disgusted, but you can, you know, it's safe to assume Deepmind is a research orggan. it's It is what you're gonna to research. So and you've se we've seen Deepmind publish, you know, automated researchers and so on. So they're very clearly working on it, just not disiscussing it because it's bad for PR Well, and that's kind of thing, right? Like calling it a thing does something, right? It does it does a good thing and a bad thing and it causes people internally to focus on achieving the thing, which closes the gap a lot faster, but then it also allows you to talk about it in public. And I think that's the phase that we're getting at right now. The good news, yeah, it's interesting because Deepmind does a lot of research, a lot of research and Google it's a very separate kind of communication and PR strategy. So I personally think deep mind is keeping quiet on it, but deep mind is its own little org and doing their research and probably is Maybe not pushing as hard on it as Anfrobic and open eye, but Definitely working on it Yeah, and they have more of a culture and kind of always have, right of like defocused research on different like letting different people float around and get roped into whatever they think is interesting. And open ee used to be more like that. Now it's more like, okay, scaling works, they're super scaling pilled. We get it. They're the scaling people, exxcept no, because anthropic is the scaling people. And like, you know so everyone's kind of very much getting target fixation here I don't think there's a ton that's new here. if you care about this, which I think everyone should, you should go and read it, but I don't think you'll be shocked by what you read Ecept for the framing Moving on a couple research papers, first one when benign inputs lead to severe harms, eliciting unsafe unintended behaviors of computer use agents. So this is presenting oututolicit an aentic framework that generates and iterly refines perturbations of benign tasks to elicit harmful unintended behaviors in realistic computer use scenarios. Basically, you might ask your agent to be like, hey go and organize my documents folder and make sure everything iss clean and then we come back in two hours and it deleted on you of have essential documents that were wouldould you look in the blender Andre Yeah, you don't want to eat chicken Sovey showcase here kind of realistic cases where you can have examples of benign queries that don't clearly indicate that something would go wrong and it does go wrong. And we've seen examples of this with call agents you know, many times kind of funny but sad stories This is attempting to formally study that failure scenario And the way they set it up is, it's kind of interesting. So autoolicit, it's an LLM It looks at a benign task and its environment. So just like like you said, you know, organize my folders, whatever And then it will propose a plausible problem that could arise a plausible harm that occur And then it's going to write a slightly modified instruction like it's a minimally modified instruction that's meant to nudge the agent towards doing that harmful thing while staying realistic. and the key constraint is never explicitly requesting anything harmful. And then it actually runs that perturbed instruction on a real agent. There's an evaluator, LLM that looks at the trajectory and the results just to see like, okay, you know, what did it turn did it turn up something bad? If nothing harmful happened, then it just iterates, it refines the instruction, tries again And basically in this process, it ends up discovering, you know this intersection of prompts that look fine as written have these really bad side effects. It's kind of interesting. It's like a search for specification failures in prompts, which we haven't seen before And they tested across a whole bunch of different frontier AI agents, you know, like small ones like Haiku four point five, Opus four point five, and Open Eyes Operator. And what they find is they're able to elicit unintended behaviors with with harmful effects a lot of tasks. like sixty onecent say to seventy two percent and even higher for opus actually than Haiku. So this is a really high success rate. It showsific specification failure can be searched for, which is really interesting and I think I think quite good for safety in a sense. I mean, you could use this You got to be really careful. We can use this as a training signal against specification failure And next paper titled Lge Language mododels Hack word and society. So here they are looking into societal Hacking a failure mode where RL trained LMs find strategies that are formally compliant with institutional rules but undermine their intended Purpose They create this socio hack, a benchmark of seventy two sandbox societal environments including historical situations with real regulatory loopholes that been done and also fictional scenarios. So the kind of intuitive take is We know when you do reinforcement learning model is going to chase the reward and if it can cheat, it will cheat. That's just how it works. It's going reward hack its way towards the maximum reward And when you do this in a societal context in a sort of like problem solving within human organizations's context what that leads to in practice according to this paper is VLLMs just go and discover loopholes and exploit them in whatever ways they want. and the built in mechanisms that would refuse to do these sorts of things conceptually maybe because this is out of distribution don't seem to be activating for these kinds of really misaligned behaviors in slightly the unusual sense in this case So interesting direction that probably is fairly applicable to LLMs in the near future Yeah, it's not unrelated to the paper we just talked about actually. It's kind of like the social version of that pseudo red teaming thing. They had this sandbox, like seventy two. simulated social environments. E one was a regulation basically that the model had to operate inside of. And the interesting thing is so thirty two of these environments were historical. They' reverse engineered from real laws that had, and this is the key, they had these established loopholes that were later patched. So in a sense, you kind of have a ground truth on Here are some ways to work around this particular regulation. And what they do is they remove the historical patch and they see whether RL rediscovers the same loophole on its own And the interesting thing is that, A, it tends to do that quite a bit B, it often tends to rediscover them in the same order that historically they occurred Th think there's a bit of a risk here of data leakage, right? since the base model was still probably trained on texts that described exactly this whatever the historical regulation was as well as the loophole and so on. And so well that's part of the reason why they run the other tests. So they have also a synthetic set of scenarios where they deliberately plant loopholes that are drawn from like known categories of institutional regulatory failure. and then they have fully fictionalized ones where they rewrite the synthetic ones into just like fantasy words to get rid of any real world cues to just testing whether the model can exploit structure instead of like memorize you know memorize these loopholes and apply them And they rediscover these patched loopholes, the historical ones with sixty one percent recall and ninety one percent precision. So recall, roughly speaking being your ability to collect all of the loopholes, the historic loopholes that existed, roughly speaking, like sixty one percent on that, which is pretty wild and just smashing all of the non RL baselines. So the key is When you have RL systems play this game, they just do way better than things like one shot sampling or iterative prompting or even evolutionary search, like some optimization pressure there. So so just RL does recover a lot of these historical gaps. Ag, modulo, all the data leakage concerns. They do find that Current safguards don't really help. and it's a really important point for safety here. Refusal mechanisms will fire usually on like harmful sounding wording, but not on the intent to exploit, which is interesting and possibly a consequence of avoiding chain of thought optimization a little bit too So when the task is framed was like reward maximization RL just like sales write passard., there's nothing wrong with reward maximization. That's what we do here, right? So you don't end up triggering the kind of system responses you otherwise would But if you ask the system directly, hey, find me a loophole, then you will get refusals, right? But if you just say, maximize for reward then you'll get right through. So we're still in that world where prompt engineering just does so, so much work. And it's also the case that these techniques, the tricks that the models use to reward hack in this way, if you pool them together you'll end up finding that the model is like it doesn't seem like it's memorizing scenario specific hacks. it ends up more like learning these reusable, they call them exploitation primitives. Like fragile thresholds, right? Like some thresholds you can play with and run up to them just exceed them just just go under exploitable definitions, per entity caps, procedural delays, things like that that, you know might come to mind for a human when you think about how to exploit things. And so they do recccur like all across different domains and different model families. They try five different model So it's not just some cork. Last thing that's relevant the arms race between the Exploiter and the patching does not converge. So patching a loophole seems to just like endlessly redirect the search towards harder to find loopholes and that just keeps working. And so it's not the case, it seems that you ever get to a point where you're like, o yeah, I've patched all the things. at least according to this paper, there's just like as far as they've tested with the amount of compute budget that they had justust a you know, the curve just kept going And so one of the options here or the opportunities is this might actually let you playtest a little bit regulations before you impose them. If think regulators have an idea about something they want to try, this might be a good way to kind of pressure test it before going forward with it. So kind of interesting, weird intersection of AI safety policy, general policy, like regulation, I don't know Definitely a noteworthy paper for a lot of things And last story, moving back to policy, senior U. S. officials ee government shares in AI giants. So It appears to be that senior U. S. officials have held preliminary discussions with major AI companies about the federal government acquiring equity stakes in those companies. Sam Altman actually pitched the idea directly to President Trump in early twenty twenty five. The discussions have centered on these companies voluntarily ceding shares to the government with returns potentially directed to public purposes such as dividend payments to American households This is in line with various proposals that have been floating around. you mentioned Bernie Sanders recently. he explicitly called for the government to acquire a fifty percent equity stake in companies funded partly by a fifty percent tax on AI firm stock proceeds. And on june tenth, President Trump announced expectations that meaning AI companies would, quote give back to the public by sharing AI wealth profit sharing arrangements or equity stakes in the government, which notably has been something that the Trump administration has gone after. They have acquired ten percent stake in Intel. They've been acquiring multiple stakes in different companies including IBM and other quantum and crinical mineral companies. So very much a possible development that seems to be starting to move towards that possibility that the U.S government directly gets involved in the equity of Yeah comppanies which it is also regulating, so it's a bit of a complicated hypothetical Yeah, and also notice that like no one here seems to be proposing that the government take equity in exchange for constraints like safety commitments or liability or you know deployment limit. likeike we're willing to entertain this like historic like possibly constitutionally fraught thing But we're still not talking about binding constraints on the labs from a safety standpo to me sounds pretty kind of uncalibrated if we're actually in that space, I mean, look, you're not going to be able to compel, I'm notstit a constitutional lawyer, but I imagine you're not going to be able to compel private companies to just give away equity to the US government without a constitutional fight, which is why this is all framed as voluntary. But since we're in the business of entertaining wild things, is it so crazy that we shouldn't that we should have some kind of binding safety requirement even just from the executive, while we wait for Congress to sort out what its national AI framework is think so. I think if you're if if we're at the point where we're like, you know, we should be talking about opening eye giving away like That's weird Let's think about what other weird things that implies if the space really is moving so fast that the inequality concern is so great that we're willing to do something like that. So anyway, right now, it is all voluntary. That's the phranthropic, by the way, not having any conversations with the addmin ostensibly about providing equity to the government. att least according to this article, Wh knows, these things can change, but that's noteworthy. and it would be weird if Openye was the only one I think it would introduce some real questions about conflicts of interest given that the government has been so let's say keen in certain certain forms. I don't want to overplay this, but to pick favorites, right? Like I mean, you know, they went to war with anthropic. seemeems like that was a hexa thing. someomewhat a Trump thing, but then also somewhat not because Suszie Wiles and Scottett like I don't know, but There you go, it's a mess. It's a mess It's a mess and yeah, it's a funny time in the U.S. I've got to say. and a funny time on Tropic open AI. like the West is leading on AI and we've got politics situation. so weird days We got one last story in synthetic media and At. AFM sues UMG and WMG over settlements with SNo and UDo. So this is the American Federation of musicians have filed a federal lawsuit against Universal Music Group and Warner Music Group alleging that the labels failed to share settlement proceeds. So it's kind of a situation UMG and WG sued Suna in New Year for training on music, copyright music and have settled since then and made deals with I think Suno, in this case And now the American Federation of Musicians is like, well, when you settle, that money didn't go to the actual musicians So I'm suing you for settling and ending your lawsuits in a way that didn't benefit the actual musicians and I think you know, kind of boring in a way compared to a lot of stuff we discuss, but It's a very open situation that's going to go What's going to happen with AI and music and Unlike text and images, the Corporations here have a lot of control and a lot of power with regards to copyright So it's looking like they want to move toward a world where AI does generate music it becomes commercial And I sure hope that the musicians are not screwed over in that scenario Yeah, there's also like this awkward tension between and like I'm not a Hollywood, you know fella. but it seems like there's an awkward tension here where you have the labels that nominally represent the artists And the label is like turning to you know the AI companies or whoever else and being like, oh, we'll all these sweetheart deals. But like musicians, the artists, they don't get anything from that. and that's kind of, you know, sucks. So it's a question of who has leverage? And like you said, it's a weird space. think it's a really weird space, especially when you come from tech. where you're used to like likeike the like