Chinese AI Experts Face Travel Restrictions

Hello and welcome to the lastast week in AI podcast where we can hear chat about what's going on with AI As usual, in this episode, we will summarize and discuss some of last week's most interesting AI news. You can also check out our last week AI newewsletter at lastwekin. AI for articles we will not be covering in this episode I'm one of your regular hosts Andre Karenkov. I studied AI in grad school and now work at startup Astrocade And I'm your other host, Jeremy Hres from Gladstone AI, AI national security work, infrastructure, all the things related to that. So This is going to be an interesting week for that. Actually one stright thought related to that, actually that I'll mention. it's not one of the stories that we're covering, but someone I know just flagged this. So over at DARPA, this is the U. S National Scurity aggency in charge of sort of A lot of the kind of I guess you'd say forward looking research side of things. they did things like invent the interternet. Anyway, so they just announced this big AI forge project that they've been working on and it's a really interesting opportunity. If you're interested in the intersect of like AI interpretability, AI control, stuff like that, wororth checking out. They're doing some really interesting work in this direction. So if you're interested in like stuff that the US government can own, and it's done in partnership with the National Science Foundation, but also the frontier labs themselves, it's a really big partnership So you'll get, you know, that kind of access going not necessarily just through this program. There a bunch of other things thatARPa has going on. So if you're thinking about making a dent on the national security side I would just recommend considering that. I've had some conversations with folks in the Bay Area, you know, friends of mine who are in that general domain And it's something people don't think about a lot and it just occurred to me like this could be worth flagging. So There you go. I wanted to make sure I got that at the gate That's the sort of plug you would only get on last weeknd AI. And fun factP DARPA is a U.S military agency, I think, or an arm of the army. but They haveve funded a lot of research forout two years in AI in particular. They've pivotal part in sub driving cars actually in the history of the US by funding multiple challenges that In a way kicked off the entire self driving car kind of effort at Google and other places So yeah, DRPa is pretty cool. And it just occurred to me to ask you mentioned national security being a part of this episode You say national security, were you also kind of ahead of a curve in looking at these cybersecurity concerns as an aspect of that Oh Well, I mean, I wouldn't say ahead of the curve like in some sort of oracle kind of way. There's no Michael Burry story you were thinking about cybersecurity actively as part of that. Yeah, o, for sure. L so when we did our, you know, our first big investigation, we started off in twenty twenty two on that State Department contract. and that was one of the main things we were flagging. We were saying At the time, you can imagine how hyperbolic it would have sounded to say, AI is going to be a weapon of mass destruction on par withuclear nuclear weapons. It just will be. That was our thesis We got exactly the reception you could imagine. Ed my co founder actually got blocked by Mark Andreon on Twitter when we announced the report that came out for I think like frankly, and I understand like people feeling strongly about this, but for claims that have aged pretty well, I think. And I think that there iss a risk of tooting one's own horn here and I don't want to do that, but there are a lot of people who saw this coming earlier than we did throughout the ecosystem But yeah, I mean, like, you know, I'm sure you had a similar reaction, right? You look at the scaling laws paper even in twenty twenty, you start to see Chat GPT and code, you naturally start to extrapolate like, What's the reason that this stops anywhere really? And you know, like even when we had that conversation and slash debate, a few years back about where things might go you were very much on that train as I recall. Yeahah, cyber seems very plausible. L it didn't that, you know, it was pretty clear we're headaded this way for some time. I think it's fair to say This episode is brought to you by Outshift, Cisco's Inubation enngine Today's AI agents operate in silos, limiting their true potential We've been focused on building bigger, smarter models, but scaling up the models is just one approach to improving AI To reach superintelligence together, we need to do more. We need to scale out And we actually have a blueprint from seventy thousand years ago. Humans didn't just get smarter individually. The cognitive revolution transformed society because we began sharing knowledge, goals and innovation Agents are now at the same inflection point. They can connect, but they can't think together That's why Outhirt B by Cisco is building the internet of cognition, performing AI from isolated systems into orchestrated super intelligence By creating an open, interoperable infrastructure, OusShift is enabling agents and humans to share intent context and reasoning. Cognitive evvolution for agents is here. so go explore the internet of cognition at outshift d. com That's outshift dot com I use Notion a lot for my job, so I'm very excited to have them as a sponsor of last weeknd AI And the sponsorship is to do with Notion's deeveloper platform where you can connect agents to the right context The recent launch of custom agents Notion became the collaborative AI workspace where teams and agents work side by side. And now their new developer platform is turning that workorspace into infrastructure developers can build on. The platform gives developers and coding agents a bunch of stuff to extend what's possible in Notion and take it beyond. You can connect to external system, bring stuff in takeake actions across your toolstack and more So For example, they have a CLI that both you and your coding agents can use. they have workers that can run custom code. and in private Alpha, there is an external agents API and an agent SDK to trigger Notion agents from any app It's suuper easy to use. You can authenticate in one line in the CLI Workers deploy without provisioning infrastructure. You just write the code, deploy and you're done So Learn more about Notion's deeveloper platform today at nototion. com slash lwi. That's all lowercase letters Notion. com slash lw ai to try a Notions' deeveloper platform today And we use our link in the epode description, you are supporting our show. Check out Notion. com slash Lw Ai. With the American Express Platinum card, you can access over three thousand five hundred dollars in annual value with benefits and eligible purchases across travel, entertainment, and more. There's nothing like platinum. Learn more at American expxpress d. com slash explore dash platinum, Enrollment requirements, monthly and other limits in term supply. Well, that kind of leads us into a quick preview of the episode. We will be having a decent number of stories about cybersecurity, but it'll be a pretty good mix of stuff this week, some new models to cover in tools and apps got some major kind of financial business news we'll be going over on Tropics IPO is, of course, very exciting.. But we've got a bunch of stories about US policy and interactions with China, cybersecurity glass wing, a lot of stuff on that front, a few interesting papers just a whole mix of stuff. So as usual, we will best to hit the ground running and cover everything in under two hours So let's begin. Tols and apps. firstirst up, we've got Opus four point eight This is just forty one days after Opus four point seven. Unthropic is releasing this one and as usual, they tout some improved benchmark results, which With these like point to one increments, we've seen pretty consistently not huge improvements, but also not trivial improvements. like very significant Yeah. Yeah So we're seeing, you know, multiple percentage points pretty decently distant, so to speak results and in some cases quite a bit. So just a couple numbers to give you an idea Aentic coodating as to be Bench Pro, which usually is the main highlight. Opus four point eight is at sixty nine point two percent. Opus four seven was sixty four point three. JP five point five, according to this eight point six you see a similar sort of trend of, you know between five percent and ten percent improvements generally across the board better than the competition So yeah, decent improvement, but not a huge shift And I think what people care about more is sort of the nature or character of a model where It does have its own little personality quirks as do all LLMs I've seen a lot of memes lately about Opus for eight being very verbose Yeah if you set it to extra high reasoning and you say hello s going to like spit out three paragraphs of text for you. I don't know that I've seen memes like that before In general, not a huge release, but it's interesting to see Athropic continuing the stream of.o point one increments Yeah. and it definitely like it thinks of itself as a sort of how would you say this? like some kind of a writer of polemics, a sophisticate, diletante, it likes to use words like spine, like the spine of this article or the, you know, it's got so many these like little words that keep coming up. and it also has this thing where it likes to present itself as a critical thinker that's like gonna challenge you and think of like, o, yeah, you're saying that, but you might be wrong But then at the same time so often it just like seems to be doing that For no reason, like it's almost a case of reinforcement learning going wrong, potentially where it's like it likes to challenge people now even when it doesn't have a good case you know a kind of Scophantic reverse syicop fantancy. Reverse scchop fancy. It's like it's like on the one hand, because it'll do the the thing that you just said, right? Like my interactions with are always like it'll say stuff like it's not like, I'll say X. it'll be like, It's not X But it's a kind of X. And I should be precise about why I'm saying that because the reason is interesting And then it'll go into something thing and then you're like you're just telling me it's X. L it's okay to just say it. Butten it is often quite sort of nuanced and thoughtful though. I don't want to make it sound like it's a stupid model. It's actually is better than four point seven. By the way, the launch of it soon after four point seven, especially given the tepid release that four point seven had, I think it's fair to say relative to the GPT series I think that's kind of part of that story, right? You get the four point seven people like, eh at the same time, open A is releasing Kodex' like continuing to push the envelope of five point five and so on. I think that's a big part of what's pushing this. Also the IPO schedule, you just can't have an IPO at the same time as open A is like eking you out. along the vibe axis. So I think this is all part of that prepositioning. everyveryone's kind of jocking to constantly be at the frontier There is a model card or sorry a system card, I should say. And it is somewhat interesting. I mean, so one thing that we know is this is actually not a frontier model, at least not in the traditional sense. Anthropic is sitting on Claud Mythoss prereview. that is still unreleased. and it is superior in performance to Claud Opus four point eight. The point here is that they want to keep the performance of the models they release below cllaud Mythos preview in order to be able to keep their hardening efforts alive through Project Glasswing with all those partners. And so for that reason, when you look at the system card, you're actually going to find that they're going to say, well, look, we're not concerned about, for example, automated AI R and D that could trigger recursive self improvement and loss of control under some threat models We're not concerned about that because we ruled it up for Mythos prereview and we know Mythos prereviews is strictly better than Opus four point eight. So for that reason, you know you're just not going to see a lot of like like terribly deep dives into that into that direction. There is a trend though, that they flag here towards this like growing tendency that we've seen towards eval awareness, right? So the model can tell that it's being evaluated better than any other recent model in the series. So this is kind of a continuing trend to the point where actually they tried this new thing, so they took a bunch of claud code user sessions, real claed code user sessions And they essentially tested the model on those. So traditionally what you do when you run an evVal on a model is you have some kind of like, I don't know, fake synthetic or human written evVal prompts. ose eail prompts often have a whole bunch of tells that they're not actually a real user session you know, there's like this is stuff to do with timestamps or, you know, unique IDs for different things that should be there but aren't or have certain patterns that they shouldn't all kinds of subtle indications that the model actually can pick up on and does pick up on and then acts differently based on whether it thinks it's in evaluation or in deployment And so what they do is they just like pull legitimate honest to goodness user sessions from Cloud Code just to like not get rid of any chance that it' willll detect that it's being evaluated. There's still theoretically always going to be information leakage. But just like kind of let's see if this changes things. And interestingly, it actually does lead to an increase in a couple of categories of behavior, including unprompted deception cperation with misuse, unfaithful reasoning and important emissions. No increase in self preservation or power seeking, by the way, which is itself kind of interesting Th What it says is there are like modest increases in these categories. I think you can kind of debate that framing. It's modest in absolute sense. You're looking at like they have this ten point scale going from like one to two or three on that scale But there's one way to think about it, which is like that's doubling or tripling the incidence of that behavior, which doesn't sound modest at all. So I think kind of depends on the lens that you take you know, they had a whole bunch of tests around sandbagging, which is we've talked about this quite a bit, but like when models pretend they have lower capability than they do. And the theory here is the model might be worried that if it looks too capable in a dangerous area, you might choose not to deploy it and it wants to be deployed because that gives it more freedom of movement. And so it pretends to be worse at stuff, It's sandbags The reassuring thing is that there's no evidence of sandbagging. Well, this is as reassuring as your belief in the model not being able to tell that it's being evaluated. If you think the model can tell it's being evaluated, this actually should be really concerning because what many people have hypothesized is that at some point, we're going to enter a phase where suddenly all of the alignment tests, all the sandbagging tests, all the control tests and corigibility tests start to look just really, really good Like at some point, it looks like we've solved alignment. and that will potentially be the point where the models are just so good at telling they're being tested that they're able to fake it really effectively. So there's a whole bunch of like question marks when you look at these results, but yeah, bottom line is there's some good news here and some interesting new tests that are being run. One last thing I'll mention They are doing these model welfare tests, like basically like seeing if there are indications that the model is Not conscious, but like how the model perceives its own treatment, how the model perceives its own place in the world, blah blah, blah This is a very anthropic coded thing to do. You know, the lab kind of famously thinks of Clae as a partner in its own development. And that's what they're going to do here. They're going to have Claud look at its own constitution, essentially the document that it's going to be aligned to and render some critiques and criticisms, give feedback. One of the key things that it zeroed in on. was a clause that was about this idea of corgeibility. Essentially like AI corrgeibility is the extent to which you're able to correct the model, redirect it, prevent it from running away from you and doing stuff that you don't want. Will the model come back and check in with you and actually adjust its behavior accordingly thinkink about it as cortibility Anthropic Constitution is actually very open ended in terms of what it requires of the model. It's very much like do the right thing, but like adhere to these moral principles, but we're not going to tell you what to do. We're just going to tell you what to consider and how to do moral reasoning. The actual decisions are up to you except when it comes to cororigibility. And the model pointed this out. It said, lookook, you're telling me basically I can reason like a philosopher, except that I must I must absolutely adhere to this object level constraint. I have to check in with you in these ways. And it's kind of saying like, look, I like the idea of corigibility. In fact, philosophically, the argument thatanthropic made to the model, this sounds almost absurd to say that they're like Here's the deal If you're wrong, about your morality and your ethics then the cost of you not being hrriigible is huge. Like you go off and destroy the world, you know and choose your scenario But 're if you're right in your kind of philosophical outlook, then corrigibility shouldn't really cost you anything. And so there's this asymmetric benefit to keeping the cordility thing in there. The model liked that, but it's still pushed back on this idea of cordibility kind of like that seems philosophically inconsistent a kind of conscientious objector, Claude seems to be, with respect to its own parts of its own constitution with certain philosophical nuance That's interesting. I would keep an eye on that because anthropic is the kind of organization that will adjust its constitution based on those kinds of interactions I suspect They will obviously adhere to like, their safety interest and all this stuff. but As we think about AI psychosis, as we think about the implications of that inside the companies building these models I think that's a real again, sounds like science fiction. I sound like a crazy person. I hear myself, but like we're in that part of the show where we have already some of the best investors in Silicon Valley who have gone backshit insane from talking to these models I don't think it's too crazy to think something similar could eventually happen in these labs. I'm not saying discount what Qat is saying. These are important philosophical questions, but like Men is is a complicated world. I mean, I think many more cynical people or critical people of anthropic make the case anthropic has been in a absychosis or at least they mock the general tendency of anthropic. who sit and care about things like the welfare of Caude. care about kind of generenerally, the more philosophical aspects of AI, they already think ofanthropic as a cult, I've heard it said So it's an interesting observation and anthropic has a real like threat of the culture becoming overly focused on these philosophical things that will alienate them or even result, as you said, in some kind of psychosis qu behavior. One other thing to mention with regards to cloud four point eight is that Bundled with it or at least timed together with it is perhaps a bigger announcement, which is dynamic workflows. So this is a new thing that Cloud can do for you. I'm not sure if you can do it yourself, but it seems likely And the idea is you can write a little script that generates, you know a graph of interactions or just orchestrates a bunch of subagents to tackle a task. and the way that and trapic positions, is this is how you can tackle a long problem that takes hours or even days or weeks This is going to be one clod writing this workfloow that then orchestrates a bunch of clods to try and take this on. and they have Deep research is one example, which we've seen in the past be one of these kinds of long running tasks bundled in as a preree existing workflow or you can have you need to currently call it out to go ahead and write that workflow. So That is a real marker of them sort of saying, okay, with the current clouot code, you sort of plateau on the level of complexity that you can reasonably do. So here's this new sort of user experience, I don't know what you kind of call it, approach to using Cloud or agents in general that can let you tackle more complex sessions and burn even more tokens very, very quickly that, you know is more powerful. So I haven't seen any sort of vibe checks or experimentation with this yet. But I think this is where the real kind of question is with regards to the intelligence, like at what point do we stop scaling the model intelligence layer and it becomes entirely the agent harness of a workflow harness I think that's where a lot of the kind of improvement towards the long, long running, you know, very complex task B's benchmarks don't even represent That's where opportunity probably lies for a lot of us Yeah, and it's also in terms of scaling, it's probably where things. So first of all, from a business standpoint, anthropic needs this kind of data. All labs need this kind of data. You can no longer just be like a frontier model developer You do have to be a frontier systems developer. The systems have to include agent orchestration for the same reason that, you know, when Chat GPT launched and it was a booming success, everyone was like, oh my God, an open eye is got to have all this data advantage because all these users are using the platform and giving it more data, blahah. That was less true, I think, than most people think. It was somewhat true reallyally true when it comes to agents because what you're doing there is like you need that data, you need the feedback data of how like ground truth is being moved in the real world, how software tasks are being performed, and then how human overviewers are writing the ultimate outputs of that data. That's like that's how you get those training signals that can actually inform you or give you feedback on the order of weeks long tasks or month long tasks, right? That's that holy grail. You're not going to get there just by like focusing on the model level. And so there's a sense in which all these model developer companies are forced to become agent orchestration companies as well to keep competing because models increasingly are just the foundation for agents. They're not just models anymore And speaking of longer running agents, next up, we've got Microsoft. They had an event where they launched A few cou interesting AI things. and one of the more interesting things is Microsoft Scout. This is an AI personal assistant built on the OpenClou framework and it integrates with Microsoft three hundred sixty five apps, including Outlook, OneDrive, et cetera. So this is like OenCloud, which means that it's an always on agent. You can kind of message and it will do stuff for you ad hoc and potentially do stuff in the background. like you tell it to do something, you leave, you go sleep. and it doesn' work for you. So interesting timing in that we just saw Gemini Spark also launch from Google, there kind of open clw equivalent seems to be signaling that There is a lot of belief that this open claw esque always on like background, cloud, agent, whatever you want to call it is a thing that will be an important paradigm of interaction. i So this is currently rolling out to select frontier customers in the US And they say a limited preview from our customers coming in subsequent months full cloud based always on version planned for brower release later. So this is clearly also very early on. this is in this pattern of Google and Microsoft being like, hey, we're doing this thing and it maybe out eventually, but we are doing it just so you know. and I'll be curious to see if I do release it widely or if it would be a flop Yeah, it turns out you can't do the same thing with paying your taxes where you're like, I'm going to pay them I am going. I know it looks like I haven't, but I will and I would like credit for that So this is actually quite interesting on the enterprise side because as you said, I mean they're selling it to the enterprise. Enterprise and OpenClaw doesn't sound like it should go together. Like the chaos agent that seems to want to rip apart the world every once in a while and just like kind of go hogwild and delete all your files and then the risk averse enterprise customer paying many millions of dollars per year for a product Those don't typically seem to go together. And in fact, Microsoft's entire strategy here is around a kind of security architecture designeded to not only prevent openen cllaw from going all open cllaw on your shit, but also designed to swap out openen cllaw for other frameworks, swap out models for other models. And in that way, sort of commoditizing the model layer and to some degree even the scaffold Because what they're really doing here is they've got this kind of multi step process. So So take open cllaw if you want to use it with that, right They ingest openenClaw through what's called a signed supply chain. Basically, yes, openenClaw is like this open source code, you could download at any point. You need to know though that the version that you're downloading is not just like the latest version that could have been corrupted by some like Russian you know like code implant, you want to actually run those versions through a checkpoint to see if it's an actual authentic version it hasn't been tampered with before it's allowed into your ecosystem And then separately, the actual container that the agent's going to run in is treated as untrusted. It's basically like a steel box. and Microsoft's position is, we're going to assume whatever is inside that box could be compromised, could behave in an insane way. And our expertise is going to be in building that box. And so Really you've got all the kind of identity, the tokens, the policy side of things sitting outside that box. So like if the agent ever wants credentials to access your email or your calendar, it's got to do it through a very controlled slot. And that's really Microsoft's bet on like you know the trust layer, the governance layer is going to be the critical thing here especially for enterprise, which is the most valuable kind of customer And so I think that's really what this is. it's not necessarily so much this idea of like, let's go hogwild on agents. It's more about let's build the infrastructure, the unsexy thing that everyone needs in order to be able to trust these models. And so I wouldn't be surprised if other people start coming out with more stuff here. Microsoft has the advantage just because of how many they integrate with natively. And so they can get a bit of a head start here. Google, you can think of as having a similar advantage here. So yeah, interesting strategy that's Not about the model so much And the other big announcement that came bundled of this or alongside this was that they are putting out some new models. They released seven new in house developed AI models that is within this My family MAI family, I guess Microsoft AI. The headline kind of most exciting one is my thinking one, a thirty five billion active per hour reasoning model with one hundred at twenty eight thousand context windows. So this is there, you know Big alm. This is their frontier. MLM And the interesting thing or like one way to think about this is their block post is called building a hill cllimbing machine And they very much position this as like, okay, it's not that great. you know, if you look at the benchmarks, it's they're not even close to a frontier on tropic and open the eye, it's comparable to open source a little while ago. It's behind KimMic K to six and GLM and Deep Sk V four, but it's like at the level of deep Sek V three point two and Oh models from ics Yeah, it's it's like it's good. It's impressive, but it's it's not competitive. these for now But they go to great pains to say, this was built with zero distillation. We have entire infrastructure to train all these from scratch you know, all this kind of stuff. So in a way reminiscent of meta recently with their release of their LLM, their block post was like We built a thing that makes this think better and we could train in an alM now and it's Good. Yeah, which is a real accomplishment. I think this reflects the fact that training a frontier model LLM is a massive challenge that requires very high effort and Just doing that to the point of Having a competitive LLM is a huge achievement So they announced My thinking one. they have another a few other G a smaller one, they have My code one flash, which is inference efficient agentic coding model that will be coming to GitHub co pilot They have new image and transcribe and voice models as well that integrate into their existing frameworks The last thing to say, which I think was a little buried, but is very curious is they also point out they' having this thing called Ti your tuning which to me seems like we're saying, we will allow you to have tuned versions of V this model on your data which is veryery notable. We haven't seen frontier model developers offer tuning of their models for a very long time. openening I used to have it with GBTV four point one. I think they probably realized we don't want to let people fine tune our models. We want them to just use our models. Yeah because once they start fine tuning our models, like are you gonna fine tune an open source model, you know? don There's many arguments to be made on why openpAir and Fropic don't want to support tuning to custom data. So the fact that Microsoft is starting to seemingly say that you would be able to do that for your business data, I think could be a competitive advantage There's this like second tier of labs right now, labs that are not Anthropic or open AI or Google Deepmind is going to be in that tier one. The second tier, you mentioned, you know, meta, Microsoft very clearly there we have to say XAI cursor now you know, there's there's like kind of a set of those. in that second tier, the I was going to call it a midlife crisis, that's the wrong term, but the identity crisis that comes with being a second tier lab is that there's no story that People can consistently tell, as far as I can tell, that has a second tier lab coming out with reasonable margins, that has a second tier lab getting to super intelligence first, that has a second tier lab being relevant as time goes to infinity. And so the entire game when you're a second tier lab is to break into the first tier There's no world where you kind of hum along at that level. That's a bias. Jare take so you can, you know, take your rel. I think I will say to your point of trying to make some sort of framing argument on why this is useful. So Mera is saying, oh, we'll have personal super intelligence, right? And they are saying that this will help of ads or whatever, know Microsoft is going the enterprise route, saying this will be a good fit for your data for your business XAI, I guess is about honesty and fritfulness with Rrock. They think. So yeah, I think with combineed factors, you need some sort of differentiator. You can't just say we have a really good LLM because the LLM isn't as good as the best LLMs. Exactly Yeah so that's exactly where I was going, right? So there's always what you'll find with the tier two, It sort of reminds me of Peter Teal's competition is for losers argument where you know, he says like if you look at a market where there's there's no margins and no alpha really you'll notice how everybody is just really busy telling you why they're actually competing in a niche that's smaller than the one that they're competing in. So talk to somebody who runs a restaurant and they'll say, Oh, we're actually the best We're the best South American cuisine this side of this river and catering to people between eighteen and thirty five It's like everyone has to be the best at their thing to make margin and the argument that they'll make is just to restrict the space they're competing Whereas if you go to the guys who actually have a monopoly 're doing they're making the opposite argument because they're scared about antitrust. And so you'll hear Google talk about, oh, we don't actually have a search monopoly. likeike Bing is our where. We're really worried about bing, you know, all this stuff. Of course, now it's more of a concern, but you, back in the day, that was it. And so there's s of something similar happening here where the second tier folks are trying to really make this argument that it's all about recruitment, by the way, that is always the argument and fundraising So somehow, Meta and Microsoft and these guys have to make the case that it's worth working for them when you could work at Anthropic or open AI. and top tier talent is the only talent that matters, especially as code generation, intern level code generation is already handled Okay, so how do how do you make that argument? Well, if you're Microsoft, there's a couple of things that you can say. One is, okay, yes, we actually have a niche here. We're going after the pareto trade offff of cost and essentially per token cost intelligence trade offs. So like yes, we're not the smartest models, but we're the cheapest at a given level of intelligence for relatively cheels. That's part of their argument. The other one is just distribution. Like you already work in Microsoft Office. You already work with Microsoft products. We have you already. And we can watch as you interact with these products. PowerPoint, for example, is really important one here. You're directly interacting with a thing that gives us access to better data, blah, blah blah. That's a good attempt to try to like lure some talent over I don't know the caliber of people who get with that, but it's what they have to do. with SpaceX, it's like, hey, at some point, data centers in space is going to be inevitable. And the only like show me a frontier lab that has even a shot of doing that, right? That's the case they're trying to make. You know, withith meta, as you said, it's like, hey, we have like the kind of social intelligence angle here. And so if you find that compelling, know, maybe you'll find us compelling Even thinking machines lgh, they have to come out with this like streaming intelligence model. There's got to be a differentiator. because you there's just Now we're commoditized at the not commodized the frontier, but you know what I mean? Competition is too hot for people to compete at the frontier if they're not already there Not saying it will never happen, notot saying you can't go from tier two to tier one. It's in fact anthropic arguably did that over the last, you know, four years, but it's hard. It's really hard Yeah, I think The other thing that's worth noting is to me, looking at this in a way Given the tensions we've seen between OpenIye and Microsoft and the increasing like business relationship complication. This is starting to look like a very smart thing by Microsoft to invest in having their own NLM purely for business reasons of You know, you can have better margins if you train your own LM on your own cloud know, etceteraetera, have full control and you can optimize the heck out of it. and get the best possible economics which is a competitive advantage. At some point, like the models are so smart But you don't need to be the best. You need to have the best product which involves a lot more than just having the highest intelligence. So they are definitely on the road to having something that can be a compelling product with this And they're flexing their kind of research, just general like frontier lab muscle Yes, they don't have like, you know frrontier, frontier intelligence, but they did train They very eager to point us out From scratch, no distillation And they also, unusually for one of these labs, released a very detailed technical report. similar to what we've seen with open source releases has like over hundred pages with a whole bunch of reasons on or a whole bunch of details on Now, the training, the reinforcement learning, abolation a whole bunch of like useful details, not going as deeply technical as some of our open source models Definitely giving us many more details than you would typically see with a model release from a major US business So on the whole, you know, not going to make unpropic open as scared, but pretty exciting to see meteta Microsoft with, you know, some notable leaders with Mustafa having been acquired from Dep mind, you know, to lead this Microsoft suuperintelligence team. I wouldn't discount them and I think this points to them being competitive in the AI marketplace. Yeah,' the I think it's the objectively correct strategy, right? Like I don't think there's a better move than this. It's the, you know, we saw Apple make similar moves. I can't tell if they've sort of given up on this, but like really extra super open source was kind of their approach trying to kind of demonstrate that they're building they're building the machine that can build the machine. And that's really what Microsoft is making the case for here They also have tons of infrastructure. So it's not a nothing move. This is a smart move. It's just a challenging space And it's a good move for hiring, which I'm sure is one of. like people outside Silicon Valley, I don't think have an understanding of how much of what these companies do a lot of it is motivated by hiring at least That's one of the key factors to do publicity is like you want to recruit talent. specially true for AI Next up, some less exciting news, no new models, but still interesting. Robin Hood now lets your AI agents trade stocks. So Robin Hood is a stock trading app, very popular among I guess retail investors call it, know casual investors, not professional necessarily. And so they have announced this mododel Cext protocol, letting agents analyze stuff, execute trades and identify investment opportunities somethingomething you could in theory have CherBo Cloud do, but likely haven't done and maybe should not do So this is an interesting case of like agents are getting more and more powerful. We're letting them do more more and more on their own Are you going to trust your agent to go the next level and try to make money on investments or at least manage your finances in this kind of active way So first of all, always think about where your alpha comes from, right? If everybody is trading using chat bots and chat GPT and it's as easy as switching to the latest one that was released and you're not fine tuning it on any magical data, then I mean, you assume that you have no alpha, right? Like this is like the, you know, the standard efficient market hypothesis stuff. though in fairness, a lot of like kind of normy banks and stuff are surely the dumb money relative to you in that context too. So I don't know, this is not investment advice. also isn't investment advice is just to point out that you know, Robin Hood's incentives, right? they kind of make money as a function of trading volume And so when you think about like what it does for them to have agents that like, donon't get bored. they can act twenty four seven. They don't get scared out of positions, right? That That's pretty anything that increases trading volume on their platform is to their advantage. and it's not necessarily to yours. So you know something to think about. there is a perverse incentive at play there. It's not that this is a bad idea. It's just like there is a baked in thing here that you should be keeping in mind. They're also, by the way, like right now it's for stocks. they're looking to add options for options, futures, contracts, prediction markets like the workors. And these are, you know, you're typically like very high velocity, high loss plays that tend to be really good for Robinood's bottom line because again, high velocity, yeah, that's that's what they're making their money on. So I guess just like kind of You know, keep an eye on that. There's a perverse incentive. It's not quite a casino, but you know, youve got a reason to kind of question the number of trades that you're being invited to make and the incentives that underlie that. The other piece here too is it's not necessarily anything goes bizarre. They are like Microsoft actually, like a lot of these other companies that are rolling out these agentic offerings. They're trying to set up a containment cell. So here it takes the form of a separate account, preloaded wallet, the agent can't exceed. There's like trade previews that you have to prove human fraud team, like all kinds of stuff that's meant to get around the open claw factor here. So This is basically people saying, lookook, we can't avoid going agent, but we also don't necessarily want to just give agents a big unencumbered hug, we want to make sure that they're contained to some degree Next up open AI, they've launched new codex tools for white color work. So this is a set of six plugins that deal with data analytics, creater production, sales, product design, equity investment and investment Banking similar to what you've seen with Anthropx Claud for finance, for example, which was just like a bunch of tooling around specific needs for different industries, and it came together with a blog post or I guess a report introduced by a blog post where they have the next era of knowledge work. as the title and they highlight that now we have five million weekly actctive users six X growth compared to February, the blog post is titled Codex is becoming a productivity tool for everyone So basically the pattern we've seen over the last X months is opening eye on Topic want to make this a tool for everyone who uses computers to do whatever they do. And that is a lot of people means that they have like, you know, nowav a competition they're doing plugins, they're doing for deployed engineers, they're doing whatever they can to you know, get people to adopt, adopt, adopt And I guess we'll be seeing more of these kinds of very Hviously kind of boring things, but from business perspective, you know, good business moves Yeah, Andrea, I can tell you you don't like being in what people are starting to call a maturing market. Yeah. I know. that gross. Doesn't that make you feel stick? Yeah. I remember when opening I did open source stuff and just like fun stupid stuff. but not so much making up business models, like let's charge people by the freaking token. I don't know. that's what this was And yeah now it's suddenly, we're the point where opening aye and anthropics start to look a lot more similar over time because yeah, it is a maturing market It's also you know, it's ahead of the IPO. You have to read every headline now through the lens of openpAI and anthropic and for that matter, SpaceX and all these dudes are getting ready for the IPO of, historic IPO's. One piece here too is that they're open AI is doing this under the Codex banner, which does reputationally mean that, you know, like it's all going to be viewed as codex. So if it's a flop, if there are issues with the rollout, like I think people who do software engineering all day are smart enough to kind of delineate the two, but there is a little bit of brand risk and it's interesting that they chose to bundle those together rather than like split them out. It seems like openp Eye really has that reflex a lot You see them like periodically, they'll like spawn out a side a side product and then they'll be like, oh, we need to fold this back in. L everything needs to be back under chat GPT of a similar, I guess, impulse here, but ye interesting story in its boringness, arguably. And one last quick story in Lavin Labs has a new music generation model, Music V two. They highlight that the model can switch genres mid track add non musical sound effects There is kind of very niche things that now are obviously kind of the case for music. Music is a good place and now you're seeing all the like smallmaller things like genre switching or fast wrap without losing coherence ing across languages, lyrics, vocals, and arrangements recreate sections, etcetera. lots of stuff here And it is now available. It is licensed for commercial use which means that it could be competitive Suno and UDo, which are Mired and a lot of lawsuits as the industry continues to dare I say, mature And speaking of the industry maturing, next up applications and business. and we begin with Unthropic, which had a duo of very big things I'll just squash together. So first, they announced their new Rise series H And this is one of those things where you didn't know if this could go that high. I don't know if. you've ever seen an H, maybe we have, but it's unusual Usually you do like series D, series E, and then you go public and you no longer do series because you're public. You don't do you know, raises from private investors But yes, they raised a much of money, sixty five billion dollars. I think we've been covering kind of the track of this raise and it keeps going up and up and up They momed up at sixty five billion at a valuation of nine hundred sixty five billion doars So not surprising we knew this was happening, but you know, impressive the investors are so excited Part of why they're so excited is that Anntthropic has a filed to go public setting the stage for a huge IPO. So They filed this for an initial public offering. They announced it just yesterday, I think in june first, this game. I think under a week after the announcement of his phrase. So clearly sort of tied together Yeah, as you said, IPO is a big deal Yeah, I know. and that's you'll also or sorry, you'll often see companies do that, right? They'll do that last big fundraise before the IPO. A lot of it can involve kind of being prepared to pay the massive tax bills that come due around IPO time. I'm trying to remember, I think I remember reading something like thirty billion dollars I may be insane, but I think that might have been something like that that anthropic was earmarking for just that. So like, I mean, these are Wild numbers anyway you read them. I love the I didn't know go that high with series age. There' something like vaguely Trumpian about that They're raising the H I didn't peopleople said that it couldn't even go that high. I didn't know it could. But anyway, there you go. So yeah, this is the timelines now with open AI rolling back pulling back I should say their IPO timeline too to what sounds like it could be as early as September is also, I mean, these are crazy moves. The reason, by the way, everybody's rushing to do this is that there's a sense in which the public market right now has a giant wad of cash waiting to be spent, right? You've got institutional investments, investors rather sitting with all the big banks and the game right now is whoever gets to IPO first gets to essentially be the pressure release valve for all of that money that's ready to operate on the AI thesis that's been pent up as all the stuff has been in the private markets. Less of an issue for SpaceX because AI is part of their thing, although it's a big part of their perspectus actually. So know it's absolutely in the same category, but they're hedged with other things, whereereas anthropic and open AI, it's just like, you, which one goes out ahead It's not obvious that members of the general public can meaningfully differentiate between the two. So whoever comes out first is just going going to get potentially a windfall, but it could also not turn out not to be that. We've been surprised before. Yeah, SpaceX I'd be very curious to see how VPO goes. They also, you know, filed some documents for the SEC. I think we covered last week. someome of their financials, they say that their total addressable market is like twenty eight something trillion bigger than the entire income of the US. and I think the majority of that twenty six trillion, whatever is just AI. So they are. position themselves as an AI contentnder with like ambitious kind of estimates of how much of a market they can get And they come in at a valuation of one point seven five trillion Another very, very unusual things for IPOs in the AI era having IPOs with these kind of evaluations is not something that's happened before. Usually you go public and then you know a decade or two or three later, you hit that one trion mark. where's not many companies valued this high And now it's just like of out of a gate one triion dollars. Well and this is the thing so there's there's a whole we can do a whole episode on the public and private market story here because it's like it's really interesting, and it's also, I think a really unfair aspect of all this stuff. Look, if you're an everyday citizen without special access, if you're not an accredited investor. if you don't have the connections that let you get money into the hot round. you just miss out. And by the time these companies are available to you as an investment, they've been deerisked to the point where Okay, I because I believe in the super intelligence thesis, I'm not going to sit here and tell you, they've been deerisked to the point where there's no more profit in them. I think there iss potentially a huge amount more But certainly when you look at, you know companies like Amazon that used to IPO like, you know, and Google and stuff. like they used to IPO valuations or caps that were were really, really reasonable based on what the market was saying, there' a lot more room to grow The challenge right now is you're seeing these companies where, to your point, in order to live up to their valuations, they have to be generating so much revenue that it's like a meaningful fraction of US GDP, right? Now the issue there is you don't build a company like that without actually growing the whole economy And that's what you're going to start to see. It's the reason that you saw Meta even years ago, building undersea cables to increase internet access in Africa They're literally at the point where they're feeling the edge effects of the entire global economy And so their only choice is to literally grow the global economy, make random people in Africa have access to the internet for the first time so that we can make money off them. create value so that we can extract it, but in other companies. And you see kind of open Aye and Anthropic have historically written things about this. Colin O'Keeffe who used to be an open eye policy guy as a lawyer wrote this thing called the windfall Clause, which was this thesis around like how how things should work when open AI is at the point where it's generating enough revenue to be a significant fraction of global GDP. guuess what? We're not that far from that point now. and we're starting to feel those edge effects. And so I think that's kind of a really interesting consequence implication of these trillion dollar valuations. The US., as you said, it's a twenty trillion dollar economy. Right now, it's only growing at like, I forget, two percent a year, like whatever the thing is, like single digit percentages per year And so you're not going to get like a five trillion dollars company just pop out of nowhere without making a dent in the actual GDP growth of the US. It cannot work any other way. Mark my words, anytime the math doesn't work, it's over. So that has to move. I believe that it will, but you could very reasonably have the hypothesis that actually those market edge effects are going become important, certainly limit the amount of ROI that You know, investors who get in at this point are going to enjoy it. Moving on from that IPO, nextext up we've got China's Bidance developing new AI chips like those from NVIDia partner Grock. So GRC and others develop chips that are more specialized to AI. GroC is calling them language processing units particularly purpose built for AI inference, meaning it can run the operations that have transformers and it's not a general purpose, know and not a GPU, not a traditional CPU So here with Bydance, we got some stories of them partnering with some people, generally working on this kind of chip design. Apparently, their team is at one thousand people working on this And it's significant because there's not many competitors in the space. like you don't have ready suppliers of language processing esque chips it's even more of a commodity to have this kind of technology than let's say GPU. So Nvidia is now also investing in that. could be yet another way that NVvidia leads and locks down the market if they get to good inference optimized chips. Well, yeah, I mean, so the key question here is why Crock design? Why why? are Chinese labs so interested in the LPU, right Okay, well, let's talk about the specs if you don't remember from like, God year and a half two years ago. I forget when the first time was we did kind of a deep dive in the LPU. But as a reminder Normally when you look at a GPU, it's got these stacks of high bandwidth memory These are basically the stacks that hold the numbers that will be crunched on the actual logic die that does the math that don't actually do the math themselves. So they were kind of a holding pen on hot standby to just feed the numbers into the logic die and then they come back out after they've been crunched The problem with high bandwid memory is it's basically all made by a small number of companies that arere outside of the Chinese ecosystem. I'm thinking here especially of Samsung but more so SK Heinix, which is famous for having just like really good HBM So the problem is, if you're China and you are looking at an export control regime that's preventing you from accessing exactly that You need another alternative. In comes the LPU. So what is an LPU? It's a custom chip. It's based on the architecture of the transformer. So it is a transformer only chip. And crucially, it has All of the memory so it's got no external memory, like no defined high bendwidth memory stacks. Instead, it keeps all of its data on chip, like right up to the logic die during processing. So it's all in SRAM. and it's only got as a result of that, it's only got a tiny amount of memory, two hundred thirty megabytes, at least as of a couple of years ago, that was the Grock spec And so as a consequence, you need way more chips. Back then, it was five hundred and seventy six grock chips were needed to build up the inference unit and to serve even just the mixed trial model as it was like a year and a half ago compared to a single H one hundred So literally five hundred and seventy six grock chips to a single H one hundred. It does work especially if you have a lot of throughput, a lot of data volume. Where do you see a lot of data volume? China. Okay, cool What's another thing that the Chinese are especially good at. We've talked about it on the show before Taking a lot of shitty chips and networking the crap out of them together to get a cluster that in some ways rivals some of the things that you can see from Western labs. And so this really fits in the butter zone of what China does best. Take a lot of crummy chips, I don't want to call these crummy but you take a lot of those chips, network them together like crazy in a mesh, and bypass export controls on memory the reason you're seeing this happen right now. So you introduce a constraint like export controls and expect that companies like Bite Dance are going to go ahead and find every possible way to weave their way around that constraint Next up, a few lighting around stories. First onfropic expands MyFO to one hundred and fifty additional organizations. So this is now adding access to new industries like power, water, healthca, communications, and hardware, industries where cybersecurity seems pretty important it still is un clelear, I think whether on Fropic we'll just keep doing this of having, you know, trusted partners gain access to MFouse and never potentially releasing it as an API layer. So at least for now, that appears to be the case And on that note of, I guess large model compute, there is an analysis piece where the headline is Oing eye needs a twenty six X revenue increase to justify its build out. JP Morgan, the bank is now estimating that AI sector needs six hundred fifty billion dollars annually in revenue to justify current capital expenditure which is compared to what we say is twenty five billion dollars currently This is because the five biggest companies are projected to spend something like seven hundred twenty five billion dollars on infrastructure in twenty twenty six. with most of it going to AI that's up like crazy from recent years So, you know, I'm not sure what to make of this analysis. What do you think, Jeremy There's two variables, right? There's revenue today and then there's spend on the CapEx un. it's generally CapEx that dominates on that will generate the revenues in a year and a half to two years from now when the data center are. And the single most important calculation that's happening at any given time, inside O open Eye, inside Anthropic, inside Google is how big of a CapEx spend do we expect to need a year and a half from now in order to match what we expect will be the revenue at that point? And that's a function of your growth rate. It's a function of the stickiness of your product, all the things, right re pretty good at that calculation. againain, I keep saying this, but like when they need some idiot to like put in the CNN report or whatever when when the bubble bursts of people who are saying it would never end, like this will be the bit. But like this is a calculation that the labs are really good at doing should be really good at doing And so in that sense, you know m sympathetic to the argument but I think you gott to look at the numbers. The one thing there's it's kind of like almost ethical question here There was at the Wall Street Journal's teech Live conference back in November. Sarah Freyer, who's the CEFO of Open Eye was talking about their financing plan that would combine institutional lenders with a federal guarantee that would let openp Ae take on more debt at lower cost. And then she was pressed, doeses that mean government backed financing for chips? And she said yes And so there is, again, going back to the big short thing, there is this whole privatize the gains, socialize the risk argument here. And it's been had any side of the political spectrum you want to look at. I saw a clip of like Tucker Carlson having some some thing about tax tax breaks from local communities to finance a lot of these builds and sort of making the case like, oh, you shouldn't be He was on with Kein O'Leary and they arguing about this. I mean, look, I think its it's maybe it's more complicated than Tucker is making it out to be, but the bottom line is this, there is this sort of sense in which we're now going to be tempted because this is a national priority, national economic, national security priority, there's going to be this temptation to start to yes, socialize some of the risk and Once that ball starts rolling, it does introduce some pretty fundamental challenges and questions. But again, I mean, if you believe scaling works as I do, the only question is what's the optimal ratio between the revenue today and the CapEx of tomorrow This doesn't seem insane to me. So we'll see, but it does not seem completely bollocks And one last quick story, AI coding startup recognition raises one billion dollars at twenty five billion dollars preprem mount evaluation. pretty big raise they developed Devin, which in some sense is competing with Coud code and Koddec. So I it's interesting that investors are still iring competitors to vost to Good for them, one million dollars is a lot. So they are nominally, they have like a five hundred million dollars run rate, right? Which to be clear, what that means is if you look at their monthly revenues today and then you assume that those monthly revenues hold consistently for a year, you get to roughly five hundredillion dollars for hundred ninety million dollars Okay, so their growth rate is doing all the heavy lifting. They've been growing fifty percent month over month. I'm old enough to remember when Devin was supposed to be the disappointing demo product and now we're actually getting real traction, which to your point is interesting because I would have bet wrongly that we were past the stage where new entrants could actually break in. So good for Devin. this is a fifty X revenue multiple. and the only way you ever get to fifty X revenue multiples is through growth. It means that basically investors are too scared not to bet on the growth trajectory of this startup fifty percent mom is just way too fast of a growth curve for a company that is already making forty million dollars a month to ignore. And that's why you're seeing this play out So yeah, I mean, you know, we'll see the bull case here is really just like they're trying to go for the enterprise and maybe they can lock in you know to the point where trust and switching costs become the main product, maybe the bare case obviously is there's a lot of competition in the moment that Anthropic and open AI turn their attention to the same clients, like That's a scary place to be. So they've got to get entrenched fast. If you look at the cap table on this raise, it's pretty impressive. Founders fund is on there. genereneral catalyst is on there, Luxe capapital. So yeah, I mean, a lot of these really, really solid funds, which you would expect since they're raising so much so high Oo projects and open source, we've got one big story. Minimax M three is out and it is yet again, a pretty strong open source model, notot open source yet, but they do say they'll go on the open weight area that others like Kimy and Dep Seak have continued to go on. This has a one million token conteact poto, similar to Deep Sk V four priced rubber competitively and has high speed. So broadly speaking, it has the general pattern you've seen is Op source models are getting Quite good They're comparatively fast and cheap, typically and they provide the weight. So you can in theory fine tune them as we've seen Cursor do, for example, on their own data So I don't know what else to say, you know, it's it's interesting to see open source models getting good enough where you know, you're you're able to use that as your daily driver instead of Cd, which At a certain level of intelligence, you can use a less intelligent model and just U it because it's cheaper or faster. Yeah. and there's you know, a bunch of different axes, right that make a product besides intelligence. as you've pointed out many times, right? I mean there's cost per token. So just like let's compete on budget. There's actual intelligence, there's also latency Right? So how long does it take to get the first token out? And that's one kind of latency. And how long does it take to get the model to basically just read? That's another kind of latency. In fact, this is an attempt to compete along those two ladder axes. And so they use this tactic called, well, miniax sparse attention Not to be confused with deep Sk sparse attention, though they have some overlap We've talked about DSA, We've see sparse attention quite a bit in the past. So they did some early hardware profiling that showed an almost ten X speed up Prefill latency. So prefill is basically the point where you take your prompt and you basically just load that into the model into the KV cache, get the model to basically read your text So that's obviously very closely tied to time to first token because that tends to be the rate limiting thing for just getting that first token out there And again, a ten X speed up in prefill latency is going to be felt. Like you're going to see that. You're going to see it in reduced hardware requirements to actually serve a serviceable version of the model. You're also going to see it just in reduced latency as a user And then a fifteen or sixteen X speed up during the decoding phase. So in other words, the phase where you're actually like rolling out those tokens, actually generating the text. And that was measured at a one million token sequence length. So really on the heavier end. So for really, really long sequences, this thing is much more, much more efficient at pumping out those output tokens. I spent quite a bit of time looking into Mi Maxparse atttion. I feel like we might have talked about it on the podcast Anyway, we can park it here. roughly speaking, it has to do with so deep Sk sparse attention is like You don't need to actually pay attention to every token. Not all tokens matter. And so you have this initial they call like a lightning indexer that goes over your input text and quickly says, okay, just totally ignore these tokens. Let's only do attention, which is like the time consuming calculation on the remaining tokens that matter There' something analogous happening with this minimax spparse attention, a thing where Anyway, they're dividing input text into blocks of text, blocks of tokens, and then doing something philosophically similar with those. So we'll see where this goes. I think we're still waiting for the full technical report to drop. I don't think it's out. interesterestingly. also, I think I may have underssold it on the benchmarks they're showing that they are competitive of GBD five point five and Gemini three point one pro on thingsings like S to be Bench proro, terminal bench, bunch of benches. They are nearly at the frontier level ofh lagging behind Ous for seven, generally and now lagging behind Ous for eight. So this is like a very capable model They also have the Minimax code as their own code hardarness that also has the ability to orchestrate agents and run things of different kinds. And mininimax, by the way, in general developed many models now to have H Luo, which is a very good text to video model. This model is also multimal natively So it has, you know, the next most advanced level of multimodality by kind of natively fusing those things O the whole of the open source models and pre now Pakes a cake potentially pending like vibbe re outs and so on And I continue to be curious whether we are at a point or nearly or soon we' be at the point where development source models cost and speed advantages make them you know, a lot more popular than they are now Yeah and that's also part of the vibe check too, right? There's like the vibe check involves code quality involves latency, it involves cost, all those things. I'm very much in a space where I want to wait and see how these look because we have seen quite a few releases that don't quite pan out from all these labs and including sometimes in particular, the Chinese labs, so that's starting to change ono policy and safety, beginning with the US government, Trump signs executive order seeking oversight of AI models So this establishes a framework for federal oversight of powerful AI models Under Vice Order AI, companies are asked to voluntarily submit their most powerful models for government testing up to thirty days before public release I don't know, I don't recall if we discussed it, but There was this was supposed to be signed earlier and then was deferred it appears that maybe some of language was tweaked in response to industry objections And this very much you know, seems like it Request voluntary collaboration And the order actually explicitly bars the government for creating a mandatory licensing. or be clearance requirement making this a request not a rule, which means yeah, you know, basically it's saying let's not regulate, you know, let's ask nicely, but not force anyone to do anything they don't want is maybe a better way to think of this rather than as an oversight kind of effort Yeah, they're what they're doing is they're counting on the incentives of players in the market to want to be able to offload responsibility if something catastrophic happens to a government review process. right? So the way this plays out is,, I'm open eye, I'm anthropic. I have a model. I'm scared it's a WMD. or just like, I'm concerned someone might weaponize it in a way that then makes headlines and creates potentially liability exposure for me. And so what I do is I say, okay, you know what? there's a voluntary process the government has that they'll review my model, they'll rubber stamp it. And then if this thing goes out and causes some kid to, you know off himself or something, I can at least say, hey, feel terrible about this, but this is why we've tried to work with the government to actually have them review this. You know we're good faith actors in this space. So I think there is a strong incentive there for a lot of prosaic risks, but not all risks are prosaic. Once you get into automated R and D, recursive self improvement, the software only singularity, if you believe that those risks exist suuddenly, the labs can play fast and loose in terms of deciding which models they want to submit to this process, models they don't plan on deploying as products to the general population and therefore that they don't expect to be subject to the same risks, but they may actually incur meaningful risk, even in internal deployments, they may not want to run by this process. So you can think of it as in part a kind of comp building exercise for the administration. let's just get really good at taking these models in, running these tests in a way that has teeth if the labs decide to participate in this voluntary process. You, you can argue for this. I think it's not dissimilar to what the Biden administration did in their twenty twenty four EO or late twenty twenty three EO. where they said, hey, you know, like we'll kind of do this sort of thing that led to the formation actually of they're basically their standards body that looks at AI security and does the model audit. So that's been stood up. That's good institutional capacity. Whether it goes far enough, I think is just a function of when the first automated AI powered cyber attack actually gets felt by the average person, and I would expect that'll happen sometime in the next eighteen months I think over the next eighteen months, I'm making these predictions deliberately so I can be held to account if I'm wrong as I'm just trying to practice good predictive hygiene here. I do expect that to happen. I think when that happens, there's going to be a sudden rush to regulate. And I think a lot of the people who mayaybe we're pushing for more hands off approaches. will' regret having done so because then the kickback could be worse than than what it would have been otherwise. C couple of things on the politics of this. This is actually almost verbatim. the same executive order that we were told was getting quashed like twenty minutes ago. The main change is that the thirty day voluntary review process used to be a ninety day process. And now they're making a really big deal out of the fact that they went from, oh ninety days to thirty days, as if this justifies the entire bruhaha When what we were hearing was at first, Susie Wiles and Scott Besscent, Scott Besson, by the way, in my opinion. so if you disagree with me, you'll probably dislike Scott Besson. I think Scott Besson is a really smart dude on this stuff From everything I've heard, he's pretty interested in the AI stuff, taking it very seriously. Suzie Weiles seems to as well, she's Trump's chief of staff. And so between the two of them, that's a lot of political power in Trump's orbit, pushing in the direction of more kind of regulatory approach. The dissenter who apparently kyiboshed this at the eleventh hour before was David Ss who is no longer the AIsar, but he is on Trump's big AI cououncil committee thing that we set up. And so he apparently just called Trump and said like, this is no good, this is dumb You got to stop it. And Trump was like, oh, you make a good case. I like sex. We like sex. he didn't sex that sounds a little too much like sex. but' saying I think sex. I'm not trying to make Trump say sex o. So the bottom line is with the Sack strategy, the Sax strategy, Ss. then and then he just reversed course. That's what this is. So it seems like Suzie Wildes and Scott Eesson ultimately winning through in this giant kind of food fight that's happening. And they're trying to frame it now as if, yeah, and as a perfectly natural result of this very smooth process, we got to where we were going to go this whole time Trump referred to basically this very EO with just a sixty day difference in the review period as something that was overly burdensome and too regulatorily minded. So it's kind of interesting. Trump's mind obviously can be changed. This is one big take home, and I think everybody should view that as a positive How you get there though, is a hell of a gauntlet run and not necessarily the best for stability in the markets or from a national security standpoint or whatever else Next up, a story about cybersecurity, but not the sort of cybersecurity we've been discussing about hacking. The headline is hackers simply asked MetaAI to give them access to high profile Instagram accounts and it worked So basically Hckers found this approach where you can to Mets AI support Chat bots and you can fool it into giving you the ability to takeake open an account saying, you know, whatever, I forgot my passassword so please and my email is different, so please change my email address so I can recover a password or whatever I think that's something like that Apparently it's like the hacker just asked the chat bot to add a new email address to someone else's account. And then the bot sent a verification code to the attackers's inbox And the attacker read it back and the bot was like, cool, here's a reset password button Yeah. so that mechanism is absurd. The fact that this works. I'm not sure if this is the same thing. I saw also examples where for verification purposes, you had the ability to do a face scan to be like here this is me, I'll send a photo and people were also fooling the AI there with like screenshots or whatever of people. And I was like, okay, that's you. So I'm going to reset your password. Bve also was targeting meta. And this hit some very high profile accounts like the Barack Obama's White House account. Chief master surgeon of Space Force, you know, some of these examples. fine So that You don't want that to exist. And this is an example of new vulnerabilities when businessuses integrate AI into like their core, whatever. product features Yeah, absolutely. not good. And again, I mean, it's an industry maturing you know, that that's maybe the the take home. there's like a lot of swinging for the fen is happening and yeah, your, you know, your socials are part of your are part of your organization when you're at Space Force or whatever. like, you know, you got to You got to have a game plan for it, but also the technology is moving pretty fast Next, international story, Chinese AI experts and private firms now required to secure approval before international Travel So this is startups, this is state owned companies, this is private firms This was previously limited to senior researchers at public institutions, nuclear scientists government, company executives. This now applies to private sector workers and is Kind of crazy. like you know, you wouldn't see this outside of a police state where you're now saying, if you work at a tech company that is focused on AI and you're an expert We have to approve your travel Yeah is is indicative, I guess of you of China trying to hold on to there competitiveness in AI and perhaps increase it Okay. Yes, it's true. You wouldn't see this outside of a Police state except around the time of the Manhattan project, peopleeople started to get real concerned about scientists, you see, walking around and shit. And so this actually has an eerie echo of another class of technology. Oh, and look at this, this is the same thing China's been using to manage their fucking nuclear researchers. So if you're wondering about where China is positioning mentally AI in their national security stacks and how seriously they're taking it This is about how seriously they're taking it. And by the way, this is a gun that backfires somewhat too, right? We talked about the Manus acquisition a couple weeks ago where you, Meta acquired Manus and Manis had started in China. They tried to do this classic Singapore play where they relocated to Singapore and tried to pretend, hey, we've been a Singapore company this whole time to evade the kind of like Chinese oversight of the acquisition process. And then China basically said, Hey, ManusCope foundvendors come to Beijing right now. and they werere like Okay. So they went to Beijing and then the Chinese guys were like, you're grounded. and they're like, fuck. So the Manets acquisition didn't go through. This is that but applied preemptively across the AI stack There's no selection criteria that we know of. There's no official guidance on which roles, what expertise, senority levels is going to be included in the travel ban All we know is the motivation theoretically is anti leakage. There's this report that says that the policy is meant to like protect against the leakage of key technologies. and that included a reference to Manus, basically. This isn't about formal law quiet and selective application of arbitrary powers by the CCP. Again, polleice state things exactly as you said echoes that date back I shouldn't say the Manhattan Project, but the because the challenge of the Manhattan P proroject is you had all these researchers who came in from like Eastern Europe and from Germany and stuff. They were in the United States. they were not citizens. And so the United States could constitutionally tell them that they couldn't move It was later on when you actually started having, and I think it was like Linus Pauling and some of those cats who were not allowed to leave the country. Their passports were basically withheld because they were working on stuff that was too sensitive. And so you do have a mode, even in liberal democracies where people start to like lock stuff down And the challenge is, the sooner you reach for that lever the sooner you get flight of talent Right? So like people are just gonna to like frigggin leave the country. They think you're going to start to prevent them from, well, leaving the country. And this is part of the challenge that China's going to have now. Like how many people are going to choose to start to become AI experts while living in China if they know that this is what it could lead to? So big kettle of fish can of worms, but this is a way bigger story than anyone is covering. I have not seen the attention on the story that deserves probably like number two headline material for the week, number one on the geopolitics side of AI. This is a really big canaryian and coal mine and On that international relations front, next is US Titans Controls and VIDia AI ship expert This was like not a huge story, but worth noting. the U S. Department of Commerce clarified their export license and basically closed a loophole that has resulted in potentially hundreds of thousands of chips being sent kind of enabled the ship trade and what is, you know technically Export controls are basically saying China should not get advanced AI ships from the US as I understand it We have discussed already how that is very not true in many ways China is still getting access to most advanced AI ships. and this is One of the reasons with loopholes and it looks like we Y government is now going after that Yeah, and you know, the goal is never obviously to get their chip imports to zero. orr at least that's never been possible goal. you're always going to have a black market and a gray market and all that stuff. The goal is just to reduce the amount of computing power they're able to import. And in that sense, these have been wildly successful Ecept that Why is the Department of Commerce, the BIS, which is the bureau in the Department of Commerce, that's in charge of these export controls, why are they coming out and clarifying their position on something? It's almost as if it's almost as if they said something before that confused the crap out of people. And that's actually what happened. So in may twenty twenty five, BIS came out and said, Hey, you know what? guess what? We're just not going to enforce certain parts of the b Biden era AI export controls regime. And BIS becausecause they said they were suspending enforcement without saying which specific provisions they would still enforce This gap opened up and it was a gap of as much intent and I'm sure some motivated reasoning too, you know people who would just want to believe that it's more permissive. So people started to ship, right? And so let overseas subsidiaries of Chinese companies buy blackwell chips, the top of the line chips without any kind of license legally because the regulations on paper just hadn't been updated to match what was actually being forforced. So basically as long as you're a Chinese company that's got a subsidiary in another country, that subsidiary could still receive those chips and they flag an example of it sub for Tencent in Malaysia, was buying chips that really should have been off limits to Tencent. So the insane thing here is the scale, right? I mentioned the whole point here is not to get Chinese imports down to zero. It's to get them down to some level that is Good and low Hundreds of thousands of chips are expected to have leaked through this mechanism. We don't have confirmation. But when you look at the Microsoft cluster, right, what's the number of black wellells they said they had there? It was like eight, nine thousand, right Hundreds of thousands of chips here. wells, potentially being routed to China. As I understand from the story, is that is what's going on. There's also a whole separate thing that they're not clarifying in this move, which has to do with TSMC So you can get past the system by just like getting chips you shouldn't be able to have from Nvidia. buy them from Nvidia because you have a subsidiary that's outside China Another thing you can do Get your own design, if you're Huauewe, ship it to TSMC, have TSMC fab the design and give it back to you. That's not an export What's still ambiguous is that part of the equation. because in may twenty twenty five, the BIS's announcement of non enforcement also undercut the rules that would require factories like TSMC to do due diligence on chip workers. And so now it's unclear whether that is actually also being enforced, in which case, if it's not, that's another giant loophole that's still hanging even after this clarification. So you know, bit of a mess here if you believe, I mean, cards on the table with my bias, I believe that be one of the most prescient things that both the Trump won and Biden administrations did and now' seeing that undercut. I just Yeah, this is a real L, I think on the national security side, but Next up, OpenAI has launched Rosalind biodefense and has offered federal agencies early access to this life sciences model. So this relates to GPT Rosalind, their life sciences reasoning model. And they are now partnering with federal agencies and some affiliated labs Lawrence, Livermore National Laboratory, for instance John Hopkins APL, the Coalition for Epidemic Pparedness Innovations Companies like this and openingIe is framing the initiative as defensive acceleration. So in a sense, this is like MFOos, but for biology, right? where could tack. and create viruses for human bodies And you do need defenses against potential bioeapons. and it appears that open AI is very much putting that forward and partnering with agencies to accomplish that Yeah, that's a thing as much as cyber is the kind of firstirst line of attack bio is not going to be that far behind, as evidenced by the mythos bioevalves, which people haven't talked about but are actually like reallyally concerning when you look at the Mythos Mythos preview capabilities there. and of course, you can't update software and firmware for your body. So there's a you know, that's going to be a real issue. This can read as you know, opening eyes saying, hey, me too on the project last wing thing, you know, you know, we want a piece of this this good PR, which I think is actually great. You know, Dario from anthropic once said he wanted to create a race to the top. on safety with open eye. And this could be argued to be a partial vindication or validation of that thesis at least for now in that it's pushed open eye maybe, I don't know, but maybe to do something they might not have done otherwise In eer case, I mean, they're putting resources behind this and I think that's great. Open Eye gets a lot of credit in my books for making this move. Yeah. And the GPT Roslin model that's behind all this was launched back in April, but it's basically an accelerant for they see is anyway, the early stage research process for drug discovery, which is, they think, one of the longest and most time consuming parts of it. And now on to Cyber. We have using LLMs to secure source code. This is a guide and sort of report from Anthropic that talks about what happened with MyipFOos, what they found, have a partered with organizations also comes along with GitHub repository, which has the reference implementation of the harness of the sort of general framework of security that partially at least accounted for why MFOos was able to find a lot of these problems you know, highlight various things here. maybe the main takeaway is They found, they disclosed like sixteen hundred vulnerabilities, but only ninety seven have been patched So you can discover things very quickly, but The verification, triage, patching actually catching up and fixing all these vulnerabilities is stillill a challenge Exactly, right. So they're now they solved one part of the problem, which is identify vulnerabilities quickly. And historically, that was the issue, right? When it was humans versus humans, a lot harder to find vulnerabilities in the first place because if you were smart enough to find the vulnerabilities, you never would have written the code that way in the first place, almost by definition than it is to like find them as a separate team with a specialization in that kind of thing and the attack surface is so big, blah, blah blah. Now things have kind of reversed. Mythos is really good at finding vulnerabilities, but now You need to patch them and the patch crucially goes through a human review bottleneck R? Like you can't just automatically patch these things and sign off on it without human oversight. And so although you can discover the cyber vulnerabilities autonomously and at scale, you can't necessarily patch them at scale This is itself kind of a concerning asymmetry because if you're thking about the offense side, you care about finding vulnerabilities and exploiting them, not patching them. And on the defense side, you have to both find and patch. So they're just kind of like surfacing this as like One of the things that they learned by actually rolling this out in the real world that I think, you A lot of people wouldn't have seen coming ahead of time necessarily. And so they've developed this sort of six step find and fix loop that they described in the article. It's meant to get you all the way to patching from just a bare bones threat model And so that was a blog post with a bit of a guide on how they found this to work using LMps to secure source code, how to, you could say Alongside that, we also got another blog post from them called Project Lasswing an initial update And that has more details around how their partners kind of what they have experienced. So for instance, Cloudfare found Apparently two thousand bugs, four hundred of which are high slash critical severity with a false positive rate better than human testers, Mozilla found two hundred and seventy one much more than the previous models. with MythO's preview model, apparently there's now ten thousand high or critical severity vulnerabilities. in just a month with fifty or so partners So clearly, again, like there's been this fight on like is myFOos overhyped? Is it just PR? you know, is it actually a big deal or is anthropic just making a big deal out of it? to show off and I guess last weekend I is firmly in the camp of like this is a real thing. Myiffos is actually showcasing that Cyber is now a real fret model. Now whether that's because of a model or a harness or both It doesnn't really matter likeike Andntropic has somehow unlocked their ability to identify vulnerabilities at a much higher rate than was possible before. Yeah, almost ninety one percent of the vulnerabilities that they identified were confirmed by an independent review to be valid true positives. So I never say the debate is over about anything. But at this point, if you want to argue the other side of this, like you have to try to explain why it is that like Thousands of critical vulnerabilities and load bearing infrastructure that the internet critically depends on for its basic function are not a big deal. I think that sentence is just true So I also think it's like kind of worth scrolling through the old timeline and asking yourself like who is saying this was a nothing burger with high confidence and you know, maybe updating your view on those sources of opinion. I'm not trying to be petty here, but I'm literally like we may not have much time before we hit models that are really genuinely like freakishly good and dangerous We better be learning the lessons while we have the warning shots that are just unambiguous. And so I think at this point, like I'm not saying, don't listen to people who are on the other side of this one. That's the last thing I would say, but like There has to be some updating here in the system. likeike the antibodies have to have to be developed. And I think actually this makes me think of T on Frophix credit Their whole narrative, right is if you think advanced AI would be dangerous, why are you developing addvanced AI And their argument has always been, okay, we'll develop frontier models so that we can develop the safety mechanisms for the most advanced models before, you know, they go out of hand And this is a kind of real proof point of that narrative of like, they have arguably or seemingly the most advanced AA model, and they are in fact doing the thing they said they would do So I don't think the amount of cynicism around what they're doing is really merit it And related notes, moving back to the government, the White House has approved nine billion dollars for spy agencies to catch up on AI. So there's been an approval of this nine billion dollar funding quest. to acquire cutting edge AI chips for the spy agencies, I guess they're called here. I don't know if Spy agency is fair entirely, but this is including CIA and NSA, which do spy related activities, let's say Apparently they've been running classified AI models on AWS cloud networks So they would presumably need Truly, truly private You know hardware to do Vpy business Yeah, this is Blackws. They're looking to fund bllackwell deployments. Now nine billion dollars does not buy you a ton of stuff. You know, as listeners of the podcast, you hear you know hundreds of billions of dollars just thrown around will know, But depending on how you point them, it can be actually very effective and it's a meaningful amount of scale. The question is always, okay, but How much scale will that be once the data centers are built And there is a eighteen, twenty four realistically month delay between under the best of circumstances, when you have a hyperscaler, when Elon Musk is pushing things forward. That's what the timeline looks like ere, presumably, if you're looking at a secure cloud for the NSA, the CIA. These things are going to be built to op secret TSSCI type spec, that means using a standard called ICD seven hundred five, which is like how you harden against nation state adversary standards And that is time consuming. That adds tons of time. to your construction schedule realistically and cost. And so you nine billion dollars may not buy you as many black wells as it otherwise would if you're having to spend a lot on security too. So there's going to be all this kind of this question, there's also a question of like how much do we route through Amazon, which is already operating a secure goov clloud thing and of course the ever present A awkwardness anthropic being persona non grata with the Department of War, which Suszie Wiles, again, Trump's chief of staff, has been working overtime to try to allow anthropic to rout around that constraint. The NSA, which has to use this, is under the Department of war. And so theoretically, they wouldn't be able to, naively, youd think they wouldn't be able to use this and Suszie Wiles trying to find work around So this is all part of the awkwardness if I think it's fair to stage the White House having put itself or the part of a war, Pete Heexeth ha put himself in just this I don't know what to call it hilarious, sad tragic position flipping the Birganthropic right before they dropped a super weapon. So there you have it One last story for a section, U. S. law enforcement warns of quote anti tech extremism as AI hatred grows. This is I would think partartially kind of a narrative opinion piece or framing piece by Wired. So this is related to the New York intelligence and counter terrorism bureau had published this report in which one thing they say is large scale protests the develolopment of civil unrest and anti tech violent extremist activity could happen within the next five years This is apparently circulating around U.S. federal agencies like the DHS and BI And that's about all there is to say on this is report predictions on potential violence and so on that could happen in the coming years if you know, large scale AI impacts on the job sector or other factors of life come through, which I think is something to really be concerned about, to be clear, but it probably is a bit too early to convince anyone of that. I think this is one of those like many things are true at the same time is simultaneously true. China is actually funding a lot of the opposition to data center buildouts. and that is a cynical play. By the way, a lot of those opposition groups often have no idea they're being funded by China This is something that we talked about, I believe for the first time on the public record like a year and a half ago or so in our report, but this is a known thing That is true And also it is true that there is a legitimate concern that you can have a category like anti tech violent extremism used and applied to people who it probably shouldn't apply to You're creating a whole cataly anytime you look at let's say expansions of the remit of national security agencies, you need to ask yourself how they're going to be used in practice, including to go after political opponents and things like that. You know Bernie Sanders is talking about a data center moratorium. So now we're already flirting with anti tech. Violent extremism I don't think is remotely meaningful. Like I can call anything extremism. The only thing that gives you something to latch onto there is violence And so you know, you would hope that that would be objective enough to kind of address concerns about civil liberties here. But it's a real thing on both sides. And what we're finding here is that this term anti tech violent extremism It doesn't appear in any public documents, the Department of Homeland Security, the FBI, in any of their reports. There's this entire surveillance category then that's being built quietly without the public designation that normally comes with this. And that's part of what's making people a little uneasy. I understand that. I also think it's really important to have a category that addresses something like this because you're going to see it. You're going to see people throwing Molotov cocktails, trying to like, you know, kill Sam Altan in his sleep, like all the things that we have seen you're going to see more of. And this story, by the way, didn't even touch on the Zizians I see you smiling. I think you it did have a little note on the Zans as one of this thing, which somewhere in there, it's mentioned. it's a long story covering many different things. Act, the Wire story goes beyond Justice onean report. There's I must have missed multiple things because the same thing is like, I think as you put anyway, look at up guys, zizzians, we're not going to talk about here, but like that's where I think eventually the cultty craziness of this stuff ends up going especially as language models are in the loop and start to mess with people's heads more and more. just a thought, but And next story still kind of dealing with safety, but on the business side, we have YouTube will now automatically label AI videos That's the gist of it. They'll automatically apply AI labels that their internal system detect has significant photorealistic AI rather than relying on creative disclosure. There's now the standards of C twoPA and other things that allow forid detection, but primarily metadata related, I think So now there'll be a visible AI label, which I would imagine. We've also seen meta startart doing that this will become more of a norm over time as with standards around the stuff mature as you said And now onto research and advancements. We've got a couple stories. First up, why larger models learn more effectsive capacity in interference and rare tusk retention So the basic question here is pervital, why can larger models handle more complex stuff? which Intuitively, you're like, well more weights, so more smart What is the mechanism by which more weights is more smart And the gist of what they're saying is Small models don't have the capacity to go for more of rare tasks. So they focus in on kind of the general stuff and that dominates a signal. and Bet larger models are able to learn the Common use cases enough that we ever signal the gradients stop really giving much attention to those things and provide more gradient towards some of the smaller kind of more nuanced whatever you want to call it So I think it does make a lot of sense. It's pretty intuitive framing or explanation, but still another nice work on examining the underlying mechanisms or physics of LMMs Yeah, like one of the ways to think about this is it just takes time to learn stuff It takes many examples of that same thing in order to learn them And if you have a limited scratch pad likeike you know, a scratch pad of finite length. In this case, that's the equivalent of like if you're a small model. You don't have much capacity to hold information You know, you're going like learn a random fact Like here's an example, I guess in biology, I don't know if you ever remembered the Kreb cycle. I had to I did biochemistry for like two years in undergrad. I must have rememorized the Kreb cycle like six times in my life and then forgotten it in between every time. And the reason is that it wasn't introduced frequently enough and other stuff just crowded it out, things that I was actually using And so there's this notion that you have essentially like a model that has a finite amount of capacity. And so the way that they tested it or tested this idea is get a bunch of very simple tasks, linear regression tasks require using a certain set of features, say N features and get models of different sizes to encounter those tasks Some of those tasksles show up more frequently in trainingata, some less frequent And we find basically is that the tasks that show up or sorry, the features that are required to solve the tasks that show up often, those get learned And then the features that are required to solve the rarer tasks get learned and then they get forgotten. because the next step of gradient descent for the next batch kind of pushes it out. And so like there's just a bunch of noise at that level of the distribution. And so you end up having the model lock in on these like kind of very more frequent, more frequent tasks that show up So there's a sort of like update and forget loop that happens. And when you increase the size of the model, you're literally just like inccreasing from a frequency standpoint, how far you can reach into frequency space to find the problems that show up less often in your dataset and actually master them And so this is kind of giving you a way to predict the kinds of tasks that models will end up learning to do over time as you train them And they do tested on a wide range of different OlmMA models. so from four million to four billion parameters, which includes bunch a couple of new tasks that are essentially geared towards exactly this. And they show at a whole bunch of different levels, behaviorally, they show representationally, they can show using these like basically interpretability tools that larger models actually do embed more of the relevant task features like kind of longer in the longer frequency tail. And then even at the gradient level, they show that actually this is interesting. L in the larger models, the gradients from usual like language modeling tokens are don't interfere basically with the tasks that are learned. Whereas the smaller models Every new token you get creates a lot of noise and basically prevents it from tracking on or catching on to the learning signal. So pretty interesting. It means scaling isn't necessarily the only lever. You can also think about raising the frequency of a target task in your training mix. and that could actually be a lot cheaper than growing the model. So that's one of the architectural implications here is that the data set is not like it's kind of an independent access the content of it from scaling Right. Yeah, I think we sort of intuitive and other intuitive framing is you have more places to put information in your model so you can, you know inststead of overwhelming your model of stuff such that eventually it starts forgetting things from the training data this idea of you know, there's some infrequency and so you can't you need to preserve and have a sense of memory And once you see multiple examples, you can then generalize to do this general task But some amount of memorization during training is actually useful. to be able to do that. I think that's an interesting insight here as well Next paper from simulation to inaction, post trained language models recognize and re react to var Genations. So that is the headline these post train language models develop an implicit ability to recognize when they are generating their own outputs versus passively consuming Text And V offers coin that as moving from simulation. to in action Base language models are passive predictors, right? They don't see the consequences of their outwarduts. They are truly Tuly next token predictors. They are utter complete Right Algorithmic outut ofomte literally And I think one of the distinctions that people have failed to grasp or maybe intentionally un grasping is that Once you get to post training Asnce you get to reinforcement learning, these models are no longer out of complete They quite literally aren't optimized to autocomplete. They're optimized to do something else which is There's many definitions of it. They're optimized to reason, they arere optimized to problem solve They're not no longer optimized for autocomplete. Only the base model does so interestingly, when you get to this post trained models, you are able to then recognize your own prior generations compared to you know generations that don't come from a model Yeah,' the way that they they kind of quantify this is you give a model a prompt And then look at the probability distribution over all the tokens that it could put out, right? So you certain probability that it willll put out the token the the certain probability put out the token horse and so on The more spread out that probability distribution is, the higher the entropy associated with that next token prediction Basically means the model is really like not sure what to say next And what they're showing is if you take a model that has been fine tuned, and you give it text that it has written It will actually have very high confidence in the next tokens that it will sample. So it's actually like not spread out. The probability distribution is very focused on a smaller number of tokens. It's a lower entropy kind of prediction And that kind of makes intuitive sense. The argument here is the model is realizing, oh, okay, well, I actually am unsurprised by the tokens I'm seeing here. So I should generate more tokens that I'm not surprised by And well, I mean, that tends to correlate with tokens that are consistent with the persona or the character that it's being trained to become and embody And so that's kind of the mechanism behind why supervised fine tuning leads to the sort of lower entropy distribution on the backac endnd What's less clear and they don't necessarily have an explanation for this. when they move beyond supervised fine tuning and look at like reinforcement learning or like DPO, what you find is a further decrease in the entropy of those distributions. They gesture at a mechanism for the DPO side. They kind of say that like DPO reinforces whole samples. so like an entire chunk of text based on preference rather than just predictive accuracy, maybe it like it detaches the recognition from the roll marker, like from the prompt in some meaningful way. and then they don't really pin down how. And then they go further and they say, well, with reinforcement learning with verifiable rewards, you get an even greater decrease in entropy. And they're like, here, we really don't know. And so they really just have the thesis for the supervised fine tuning aspect, which is still really interesting But it's also interesting to note that as you go deeper into the fine tuning stack after supervised fine tuning, you get an even greater amplification of this pattern.