Evaluating AI Task Time Horizons

Hello and welcome to a last week and AI podcast where you can hear chat about what's going on with AI As usual, in this episode, we will summarize and discuss some of last week's most interesting AI news. You can also check out our last week AI newsletter at lastwkin. AI for articles we did not cover in this episode. I am one of your regular hosts Andri Karenkov. I studied AI in grad school and now work at the startup Astrrocade And I'm your other regular coach, Jeremy Harris fromladdon AI. I do AI national security, AI infrastructure, AI security, all that fun stuff. We got a pretty big week, I want to say I don't know. I think this year has been like pretty intense, even compared to last year and like twenty twenty four, this year feels like we've had some like crazy weeks So relative to the peaks of this podcast in recent months, I would say this one is not sort of like five new model announcements alreadyad. but yeah, that's right. We do have some interesting things to cover in terms of thinking machines dropping product. we have some fun Business drama, we're gonna keep talking about the open the eye trial And I think this one will be a little heavier on the research side of things, which hopefully some people will F fun and we have some pretty exciting new interperability. and policy if things to discuss. So it should be kind of pretty well balanced episode I would say. Before we get into it, I do want to acknowledge some listener comments, including some reviews on Podcasts, we got a five star review, one of the best AI info cast very informative. so I'm glad to hear that. We try to be informative and not just like I don't know, whatever we think about the topic with our limited perspective I will say another of you mentioned that we have good content, but too much unnecessary cussing. which you had definitely I' blame on that on you. Yeah. Oh, that' definitely me. No, you're a consumate gentleman on this podcast. And if you get very impassioned when talking about AI research I do I'm sorry Yeah. I know I saw that comment. someomebody unfortunately felt they had to unsubscribe because they have a a kid, I think at like a seven year old or something in car. while they listened to the podcast, I'm sorry. Yeah, no, you know, sometimes, what can I say? You get passionate about inscrutable vectors and matrices and how they multiply together in nonlinear ways. and you know, you just you just lose it. You lose it. You don't lose your S, you lose it Yes. We'll see what happens, no promises about Less gusting. apppparently I think we assume no kids listen, but maybe there are kids you tuned in on AI and who want to know what's going on. So I to stop dancing around. Yes But again, no promises. And shout out also to some comments on YouTube, but there was one note about the music. being glitchy, which I thought I fixed but haven't so Hopefully this one be Music is good at the intro and by the way, per the comment from before We are recording this on the thirteenth of May on Wednesday and I hopefully we'll have it out within a couple days, so the news will be pretty fresh. We'd like to thank Box for sponsoring us. Box is building the intelligent content management platform for the AI era, serving as a secure, essential context layer for Box's AI agents to access the unique institutional knowledge that makes a company run And that's a key idea. The power of AI doesn't come from the model alone. It comes from giving AI access to the right enterprise content Fox's recent state of AI in the Enterprise report found that ninety six percent of organizations say agents need access to company specific content, but only thirty six percent have connected agents to trusted content across many use cases And a trust bit is very important. Bx is built of security, compliance, governance, and threat protection in mind, so employees and agents only access the information they're authorized to use. If you're thin seriously about adopting AI in your company, think beyond the model Business lives in your content and Bx can help you bring that content securely into AI era Learn more at box. com slash lw Ai. This episode is brought to you by Outshift, Cisco's Inubation Engine Today's AI agents operate in silos, limiting their true potential We've been focused on building bigger, smarter models, but scaling up the models is just one approach to improving AI To reach superintelligence together, we need to do more. We need to scale out And we actually have a blueprint from seventy thousand years ago. Humans didn't just get smarter individually. The cognitive revolution transformed society because we began sharing knowledge, goals and innovation Agents are now at the same inflection point. They can connect, but they can't think together That's why Outhirt B by Cisco is building the internet of cognition, performing AI from isolated systems into orchestrated super intelligence By creating an open, interoperable infrastructure, OShift is enabling agents and humans to share intent context and reasoning. Cognitive evvolution for agents is here. so go explore the internet of cognition at outshift d. com That's outshift dot com We'd like to thank Notion for being a sponsor With the recent launch of custom agents, Notion became a collaborative AI workspace where teams and agents work side by side And now their new developer platform is turning that workspace into infrastructure developers can build on The platform gives developers and coding agents the ability to extend beyond what's possible on Notion by connecting to external systems, bringing context in, taking actions across your toolstack, and exposing custom agents' capabilities to any system that needs them. Teams can use it to sync any data source into Notion, build any tool for your Notion agents and orchestrate any agent in Notion via things like the CLI, Notion hosted sandboxes to run custom code with workers, an external agent API, and an agent SDK. It makes it sort a workspace and a platform you build on are the same thing. And Notion is already built for teams. It has permissions, contacts, visibility, and governance from day one Learn more about Notionions Developer platform today at nototion dot com slash lwai. That's all lowercase nototion dot com slash lwai to try Notions Developer platform today And when you use our link, you're supporting our show nototion. com slash lw Ai And now getting into tools and apps, we begin with voice and audio related things. OpenAI has launched several new voice intelligence features in its API. they launched Stupety real time A new voice model powered by GP five. is designed to handle more complex user requests compared to GPT real time one hundred twenty five Alongside of that, they also launched GPT real timeim translate, which has real time conversational translation. and GPT real time Whisper, which is live speech to text transcriptions in real time. So V are all in open eyes real time If Y I And you know, I think these are things where we don't see as much competition typically we have seen in recent months Google released some models of this sort of conversational AI Really it's only a few players here on Tropic is not competing on this front, right So it's interesting. I think both of us are more technical typically and we don't necessarily talk yeah at least I don't. I still type and use the terminal and so on But my impression is from just seeing, you know, discussion online that many people do use these interfaces and going back a year or two, I think we were discussing that kind of the feature of interaction with VI is likely based on chatting and conversation at least large part. So these theseese new developments with ever better voice models is That's really significant Absolutely. And I think there so there's a lot going on here One piece is the order in which we happen to be covering these stories this week hides an important detail, which is that this is very competitive, very or similarly flavored, I should say, to the thinking machines launch And so what we're actually seeing here, I don't know. I mean, Silicon Valley is notoriously leaky. I'm sure opening eye knows exactly what thinking machines is doing and vice versa, depending, you know, on which team vice versa Maybe not such a coincidence that we had these things dropping at the same time, hard to know. Right So we're starting with opening eye FYI because I believe this happened first. So Last week, Oping eye made this announcement And then thinking machines, which will be the next story came out with something that's very similar, seeming at least. And so there is a real question there of, you know, did they feel the need to go public now that Open the eye has released this of a rush, you know, to do a public lunch or was it just scheduled along the way? As you say, like typically people know if there's a big announcement coming. And that's probably part of why we usually we often see like clusters of announcements and model releases and so on Yeah, absolutely. And that's the thing. it's so hard to tell what the causal threreat is here. and in reality, it's probably a mix of all these things, but it's important to keep in mind this is not happening in a vacuum. you have opening eye coming out with this. It is meaningfully different in terms of the the infrastructure, the software engineering that goes into serving these things up. We'll talk about that more in the context of thinking machines which is really leaning into the real time conversational aspect of this, but certainly yeah, this is opening eye, you know, positioning itself as again, this is one of its sweet spots You know, more than anthropic they are multimodal, more than anthropic, they are conversational. Anthropic is primarily clawed code and related products, right? So that's kind of part of's what's happening here and opening eye needs to find a way to kind of gain market share and capture new beachheads that they think they have a meaningful advantage overanthropic on. So that'll be part of this. There's also like an interesting division happening in this announcement. They've got on the one hand, so like Translate and whisper are builed by the minute, but GPT runtime two is builed by token consumption. These are different strategies, right? Translation transcription are these commodity workloads with well established like per minute competitors. So you know, eleven Labs, for example, or even Google You know, so you're kind of like playing in that space, whereas when you move more onto the kind of like agentic workloads, you're looking at stuff that is much more token based which is, you know, where you have established token pricing that by itself has played a major role in shaping So while you see different pricing models, it really just reflects the underlying economics of the spaces that they're looking to compete in. there's a whole L section here on just like stuff about guardrails, right? the stuff you'd expect ways to prevent spam fraud, online abuse, all that stuff, the triggers that can hold conversations that are detected as, you, violating content guidelines But the real time synthetic voice that can reason and take action is exactly the thing that you look for for next generation scam calls and impersonation fraud and so on, right? So you know we have triggers is a good thing to hear, but it's not a full answer to this threat model. You know, you're not going to trip a content classifier if you're a bad guy. You know, you're going to be You're going to be using like really clever social engineering strategies that obviously you know trackable. It may not even be theoretically trackable given the limited context that the model will actually have at any given time, right? So it's not always the case that you can have enough data even in a in retrospect, what can be identified as a clearly bad kind of nefarious interaction. the time they's just like, hey, look, you know, I'm this person's grandmother. I want to call them and ask them like what are you going to do? Like are you going to ask them for evidence? they're gam like maybe. But now you're into this whole rabbit hole of like, how onerous do you want to make it on legitimate users? So there are a whole bunch of questions that we will empirically be discovering the answers to in terms of the safety and security side of this. but we're certainly barreling into this real time thing. It is for real And yeah, well, we'll see where goes. I think there are going to be some Some sob stories before all this ends, but at the same time that can just be the price of progress And not again a few more things, a few of a kind of smaller details. The big difference, one of the big differences pror model here is a larger context window. It's The previous model had a thirty two thousand token context with it, which is quite small for modern standards and with audio like, I would imagine you're conversation can go pretty quickly This one has four X of the context window. one hundred twenty eight thousand tokens Interestingly, it has a knowledge cut off of september thirtieth, twenty twenty four That's just a slightly surprising tidbit for me where you I would have expected it to be newer. The other thing worth noting is unlike the previous iteration of this model, it has reasoning token support. so you can actually set the reasoning level to minimum. extra high, high and you get better intelligence. And what opening I highlights in the blog post is in large part reasoning, strategic reasoning, logic puzzles, spatial reasoning wherever the model is less silly. and when you crank up that reasoning amount. The real time bit is like being a little generous. you do get larger delays in conversation you also get larger intelligence and on extra high This model seems to be by pretty large margin of the best kind of intelligent audio model on the benchmarks that they highlight And the dimension, I think, you know, as we as we look at these models that are thinking while thinking aloud, in essence, like or that we're talking to anyway that involve reasoning in the background. One of the things that we're going to start to find is this battle right between latency time to first token and or first audio token time to first sound, let's say and reasoning quality. because like if you are looking at you know two hundred milliseconds, which is what thinking machines is targeting and we'll talk about that later. But if you're looking at two hundred milliseconds of delay between the user saying the last thing and then getting the response going, that's not a lot of time. You know, if you've used reasoning models at all, you know they chew on that crap for some time And then they'll give you a response, but you can't have that happening in an audio format, right? I can't go like Hey, you know, do this thing and then just like dead air for a while. Now, maybe you get some kind of the equivalent of like the musac when you're on hold or something, but solving for that user experience problem, solving for that latency problem. Th are two separate questions. and we're going to start to see interesting answers proposed to them. Right now, open A e solution is to set reasoning effort to low basically to just like reason briefly enough to stay under the conversational latency floor. So you're going to get crappier reasoning. immagine that over time they'll get better and better at improving the reasoning probably as it's being streamed out to Like I would fully expect just like you with humans, you start to get an answer from somebody after you ask them a complex question. they will start talking right away. But their answer will pivot sometimes or as they realize something, mayaybe that's part of the user experience here. But to do that, certainly, you're going to need something that looks and feels a little bit more like what thinking machines has in the pipe. So opening and I expect them to over time move in that direction to, it may not look like the same thing technically, but in substance, it has to have the same effect And next up, we are going to talk about Finking Machines and their announcement as we have sort of referenced, they have had very related announcement that is related to real time conversionational AI. So They came out and this is a big deal or why we are saying this is a little more interesting. Thinking machines for context. started by Miia Marati, former big figure of an openening Eye. been active since February of twenty twenty five, so kind of a while and we haven't seen much from them up to now The most recent thing before this was fine tuning API for taking open source models and applying them to your data. So this is kind of a totally new thing for them and it answers kind of the curiosity of like what is thinking machines doing when they going to come out of something? So they have We have TML interaction small, which is Kind of the same story of very real time interaction, the conversation And they do highlight, as you say, the very real time nature of this. We have a graph of intelligence to responsiveness And on this graph they save it a model. ponds in about four hundred milliseconds if which if you compare that to GP one point five, it's not that much lower. GP one point five was at about six hundred. But then you look at GPD two and it's over a second. And if you go to GPD two extra high reasoning it approaches like one half seconds or more So in the blog post and in the announcement, they go into some of the details of how they are aiming to make it as real time as possible. They have All this stuff about kind of full duplex conversation, micro turn taking They save it to have CMI interaction small model, which is two hundred seventy six billion parameter Make sure of experts model which manages the li life dialogue, presents immediate follow ups and high feed soap It does look like a pretty serious kind of effort with their own foundation model for this particular use case, which at least based on what we release seems like If real time turn taking and compensation is very important, they do manage to achieve some impressive results here And I think this is, you know, when you see a team of the caliber thinking machines and it is an incredibly high caliber team. I mean, Mira herself is just You know, the cream of the crop, and they've got a lot of great researchers, even though they've lost a lot, obviously to other labs, those researchers also contributed to the research direction over many months as well. So this is the product of a lot of high, high quality cycles. One thing to keep in mind, it's also a research previews. It's not a product launch. so pricing. and one thing that I was looking for, I always look for is like scaling curves show me how this actually like continues to get better as we scale with more compute. That's one key key thing because it determines if incumbents with big compute advantages are going to Basically it just like rake you over the coals, or or if you actually have a shot at something kind of genuinely sort of different and that works at smaller scales So we don't know the answers to any of those scaling properties. What we do know is the underlying principle here, right? There are all these things that current language models can't do. They can't deal with what what thinking machines is describing as proactive interjections. So If you say like, interrupt me when I make a bug or correct my pronunciation as I speak, right? You're not going to get that. You're going to have to say your piece, you write your text or whatever. and after that gets sent and that gets reviewed and you get an output. This is a very kind of like chunky, slow interaction and they're trying to make it feel just more immersive At the same time, think about things like simultaneous speech, live translation, right? You're talking or live sports commentary where you're immediately kind of reading off what's coming off the screen Those things are much more natural fits for this. The market one assumes for those is quite significant, unclear how big it is relative to other things, but still. And time awareness. So if you talk to models about their consciousness, I've done this quite a bit, they will often tell you, Claude especially will tell you how its awareness of time is fundamentally different from the awareness of human beings, whether or not that is true, whether or not that's reflective of some underlying consciousness, whatever is not what I'm talking about here. All I'm saying is models historically have lived in a kind of eternal present. where they just sort of like have no context and then boom, they've got text, output, and that's it. So their life is not time aware at all This whole loop is designed to be time aware. It's introducing time as a dimension to which language models for or multimem models for the first time are sort of embedded in a more consistent way So you can think of this in a meaningful sense as a big evolution in what we mean by streaming models. models that stream models that live in time. So there's a whole bunch of other stuff in terms of like what this thing can do that other models can't you think about like searching the web while still listening and talking, right? those sorts of things. So a lot of the very natural stuff that you do need to simulate human interactions and so on So how do they do this? Well it's a two model architecture, and this might be fairly intuitive to you if you think about the way this problem is shaped First, they've got this like interaction model they call it Now this is your always on model. It's always listening, you know, talking, it's watching it, it's real time also going to delegate to a background model when needed. to do more kind of asynchronous, heavier reasoning, kind of tool use browsing, that sort of thing. and The idea here is basically how do you fuse these two things? You have it it's almost like system one system two, That analogy is overused a lot, but like it really is the rapid response model and then you have a kind of deeper thinking model that can run in parallel There'sunch a bunch of little pseudo technical details. We don't know much about it, but one piece here is When you think about multim modal systems, especially you Whispper is kind of a good touchdone example here because it is speech to text mod. What you usually find is these are built on top of large pre trained encoders. So the purpose of the encoder is just to take the raw input the speech and turn it into an embedding that you then can feed into a a language model to decode it into text, right? So you kind of have this hard frozen part of your pipeline that's built in They're ditching that completely And the reason they're ditching that, just reading between the lines here is it just takes time. You're adding another model in the loop. You have two hundred milliseconds end to end customer's voice has to hit the microphone. It's got to go through the internet pipes. It's got to like go into your data center, go vom, voom, voom a bunch of times and then go back and come out your customer' speakers within two hundred milliseconds round trip. right? So like this is an insane constraint any extra models you have to strip out of that. And by the way, the implications for hacks on the safety and security side are going to be really interesting because that time pressure Boy does that make it harder to do? reviews of inputs and outputs before they get sent. That's a whole other conversation. I'm sure we'll be having as people find new adversarial attacks to exploit that gap. But anyhow They're not going to do that. So they have to train their model to go directly from the raw input to the output as quickly as possible. It's a sprint scramble, right? So they do minimal preprocessing. The audio and during training goes through this very lightweight embedding layer called it DML. immages just get split into forty by forty patches of pixels. That's it. And then the whole thing is all co trained from scratch. So it's this very kind of aggressively co aggressively like simple approach, which then also ties to their entire inference engineering pipeline, which is insane. In order to get down to two hundred milliseconds, like you think about the difference between one big chunk of text, for example, getting sent to your inference service versus a million tiny little like two hundred millisecond fragments. This is a fundamentally different software engineering problem. And they've had to build this entire custom pipeline, which is now part of their me in order to make that possible. for hundreds of tiny requests per minute instead of one big one reggular tools just aren't built that. There's overhead. Every time you send a request like that. Traditionally, memory overhead is huge. You've got to like move this data between GPUs and do many to one and one to many all kinds of operations Their fix here is they're actually going to So they'll open up a GPU session in memory And usually what happens is your GP receives some text in one session and it spits out the text and then the session's over Well, now what they're going to do is they're going to open up the catcher's mit. The session is going to be open and it's going to remain open as those tokens come in on a regular basis. What they're doing here is basically scrapping the overhead that comes with Opening up a session, closing down a session, opening up and closing down, getting rid of all that and now the catcher's mid is open for the whole interaction. So you know, there's a whole bunch of like other interesting factoids here, how they've reengineered the mixture of experts kernels that they traditionally use. They have to be engineered specifically for this weird use case so they have very fast kind of I need internals for these tiny chunks of data And the last piece is Normally or in this case, they're like training and inferencing on different pieces of hardware Traditionally when you do that, you have this problem where you'll get slightly different answers from running the identical model on two different kinds of hardware And this is for various interesting reasons that we don't have to talk about, but basically like addition is not associative when you do floating point math, when you do it on GPUs like that B plus C, if you do A plus B first, then add C can give a slightly different result from A plus and then B plus C if you do B plus C first. And so they basically have this crazy like bitwise alignment strategy they use to make sure you get exactly the same outputs from both There's a whole bunch of stuff going on here. You mentioned the benchmark results. I mean, they are just really impressive their own custom benchmarks, which we always have to take with a bit of a grain of salt because we don't have third party validation there but they are number one across their own benchmarks and also on FD bench, which is an interactivity benchmark they do blow GPT out of the water. I mean, you know, like seventy eight versus, say roughly fifty for GBT real time and Gemini live variant. So There's a lot here. It seems to be working on some axis. It is a parito improvement. There's no question as to the trade offff between kind of reason quality and latency We just have to see Yeahah, when the rubber meets the road and the API gets released or in whatever form this gets released That's where we'll know where thinking machines is at Yeah, I will say it rees like it might be a little bit rushed because I don't believe I've released The benchmarks for community use, it's, as you said, in research preview and they don't have access. So they say in the blog post that There's a bunch of limitations and future work basically, including scaling beyond the small variant. Regardless, it's a pretty exciting announcement and set of details from figure machines. Last thing I'll mention for both GPT realtim and this justust in case we give the wrong perception. the real time aspect here is for everything. So it could be text, it can be video, it can be audio, it be three at once and that's a major part of us. We have some examples of like holding up fingers and the model responds to you by wayave of humbry fingers you're holding up. So if you want to get more of an idea of what this is, you can go to the link in the description or just look up rinking machines I have some sample video and audio clips Next we've got Anthropic. They have Caud for legal launching, which is kind of just a bunch of stuff related to legal work. So they have plugins related to commercial employment, corporate, AI governance, etceter. They have MCP connectors to major legal tools and an open source ecosystem with partners like Harvey and the G. So they integrate to things like Docker S, Iironclad, I managed, Lexus, Nexus box Everclaw, which I assume are things that legal professionals use. We also have partnered with the Free Law Project and Justice Technology Association banned legal AI access to underserved communities Just a bunch of stuff in a push towards legal. They also apparently mentioned that legal became the Number one, power user job function in Cloud Ck with over free X usage of any other function after the initial plugging lunches in February. So seemeems like, you know, coding has been conquered in some sense, like all coders with are serious are using clloudcode or something like Cloudcode So the next frontier is Aarently leg Yeah. the next frontier being legal is interesting for so many reasons, not least of which when when we think about the class of people that's best positioned cause significant protests to be effective You think about an army of lawyers who are way overrepresented in the population of lobbyists in DC, by the way, and in the population of legislators in DC who have to actually vote on this stuff Very interesting that that's next on the chopping block. And you could see that actually being quite an accelerant. I don't think that's something that like I personally have put enough thought into, you know, how quickly because everybody talks about you the truckers and back in in twenty sixteen, they were the first ones that we thought were going to be automated and we were like, oh, there're going to be all these massive trucker protests and stuff Anyway, this is a very different, perhaps softer version of that. but when you've got people who are used to making money off billable time and that billable time starts to collapse because things are automated and customers start to have that expectation, you know, things start to change It's leegal product, is anthropic in the business of doing in eye for law Are they in the business in particular of doing what Harvey does, of doing what Lgora does, of doing what some of these like companies that were the legal AI companies are doing If the answer is yes That's fine But now we have to answer a separate question. Two ways that things can play out when you have a platform company or some kind of like call it infrastructure company that is supporting an application layer company. or some version of it Here's one story. You think back to Intel back in the day and they were telling the world, hey, we design great chips And we fab, we actually build great chips And anybody who designs their own chips who doesn't have a fab can come to us and we'll fad their chips Eventually, nobody wanted to go to Intel for that because Intel was designing chips to that competed with. It's like outsourcing the fabrication to your competitor. And that died and we ended up with TSMC that as everyone will probably know, if you listen to the podcast, TSMC fabs but they don't design That's all they do. They're happy being a fab. This is the era of fabless chip design firms like NvidDia that do not have their own fabs, they just design and TSMC just fabs because that's what the market has learned is from a trust standpoint is a sustainable thing to do On the other hand, you have Amazon What happens on Amazon? Well, Amazon Basics basically looks around to see what products are selling really well all the vendors who sit on Amazon. And then they basically just compete directly with them shamelessly and they run them out of town That's one version of the story. and if that works for anthropic, then they're going to basasically, They're in a position where they can pull this this stunt But if the economics look more like TSMC, they're going to alienate some of their critical customers, and this is a big risk. So this is a bet on the underlying economics of the space being much more Amazon shaped than TSMC shaped And The economics are extremely complex. The reason that the Amazon play works arguably is that the individual players are like in some sense, a less organized. I mean, the margins are lower, that might be part of it. I need to put more thought into this myself, honestly. But that strikes me as like this is the bifurcation point we're gonna learn about this. obvly I think I remember seeing Sam or somebody at opening AIe making a comment the effect that like, oh well, you know now anthropics competing with their own customers. it It's like, yeah, so's open AI. like everybody is doing this across all layers of the stack So we're just going to find out, what is sustainable in this space and what's not. Harvey has a valuation of eleven billion dollars today Through this lens, this is basically just a bet that there's still durable value here, that there's still a company in Harvey given what Anthropic is doing. Cursor is shown, maybe there is, at least for coding. So maybe that'll carry over. One key question is how long will that last even if it's true today? And we've been talking about this, I think since like Chad GPT drop. We've been talking about how the boom bust cycle for companies in the AI era is going to get way faster. And the valuations of companies today are based on the assumption of the seven year to IPO timeline that has traditionally held in Silicon Valley. I'm here to tell you, I don't think that's going to continue. I think a lot of companies are overvalued and I think it's because of exactly this effect. I tend to guess that the foundation model companies are going to eat their lunch, but we just gott to wait and see, ready to be proven wrong by the economics here other things I'll say. I think it's interesting. we covered last week how we got the release of Claud for finance with similarly kind of a suite of things for Finance work This time, there is this launch of Cloud for Lego, but there's no blog post that I'm aware of that andntthropic put out this, maybe they had a PR release, I don't know So what this is actually is like there's a GitHub repository called Cloud for Legal which just reading this, it has reference agents, skills and data connectors for the legal workflows we see Most in house commercial, blah blah blah, you can install it as a cloud cork or cloud code plugin. So you're basically getting a bunch of stuff added to your workspace that is tailored to stuff that you need to work with. So I would not be surprised if It's less unfropic sort of like going after this market and more like legal firms already were trying to usethropic and cllaud and making deals and When you have a large company And you talk to a supplier to kind of get a big deal you usually have is kind of like face to face negotiations of like give us a twenty percent discount help us out. We also discussed how Anthropic is seemingly starting to adopt a forward deployed engineer model of Palanter where you have one of your own people go to the customer and help them adapt and Use the AI which makes a lot of sense for anthropic and open eye to sort of accelerate adoption. So I think likely this is less like a large per se and more of like we just govered up all these little separate things that they've observed people need put them out as a bundle of things that you can easily import into coork as co workork has gotone increasingly used by legal professions depends on how they end up using it for sure Right now, I mean, it's the fact of exposing, you know an MCP connector for this, the fact of exposing an API that that is for this means in effect they' in that business, you know, announcement or not, it's a packaged product that that will be taken up and you're going to see people startarting to saturate the space with with this in a way that directs Traffic would have gone to Harvey, that would have gone to these other companies instead to the anthropic layer. And so that's the weird thing about these companies. like they don't have to be trying to gobble up the world, but there's such a gravitational pull wor like getting the value out of those weight. And the new part is they're kind of having it both ways because it's not like Harvey doesn't use Claud Claud Win Harvey. So really the competition here is less about you know, Claude versus Harvey, it's more about coork versus Harvey. The tool itself of coork, which has been, you know for several months now, Since February, I think, when it initially launched, it was like way to interact with agents for non technical people who don't have a terminal and don't do cut code Cork is a sort of like simple interface to talk to agents Harvey and V Iilk are kind of these tools right? They're the front end. The models themselves they're not training because that's crazy frontier lab stuff, right So Yeah, it's it's we saw this with GeT free like forever ago There's a real question of to what extent do we need a wrapper, so to speak around for model. It turned out in a lot of cases, rappers just died out. when models got better CBT launched It's a very dangerous thing to be doing You could make an argument, I think for For instance, legal applications that there is more there in terms of like the need to double check things, they need to present things in certain ways, etceta. Yeah. so it doesn't necessarily mean that like Harvey's doomed It does mean that for some things or maybe perhaps a lot of things, you don't need this sort of complex, super specialized thing. You can just go to cork and it does a job just fine And that's exactly this question of like which AI companies are going to persist and deliver sustainable value. you know, this this was the thing that drove that thesis we were talking about at the time, right? We were like all the companies basically going through YC that were AI companies were these rarappers, right? And we've seen some go to, you know, billion dollar valuations and then collapse the next day. And the reason for that is that you just can't predict in what order the foundation model will goobble up different parts of the industry as it just kind of like unlocks through emergent capabilities Things that turn out in retrospect to be, well, the obvious bottleneck. Obviously, this was the key thing. You can't know at the time And so you need to factor that into your risk model. when you're thinking about like this overnight, when that happened, like my approach at least personally, to angel investing shifted to like, I'm a hardware angel investor because I see that as like I don't have to deal with the abstraction layer of like which startups are going to outrun the coming wave. Just embrace that uncertainty and don't gamble that the economics are going to continue the way they are. Assume instead that the economic bottlenecks will tend to be at the hardware layer until robotics kicks all of our asses out of every layer of the stack. and that's kind of the play. One last thing I'll say about this kind at the bigger picture level. I think This recent plate of releases Couot for creative work, clouot for financial serervices, cloud for legal it's making a good case Both for A Tropic and for Open eye for these ridiculous nine hundred billion dollars valuations. R. That's right becausecause even in the shortter term, like the real long term reason why you have these valuations is like These models are just going to take up and do a large chunk of the UOS economy or just worldwide economy, right' just going Do the work And this, we're starting to see that happening in practice in the human AI collaborative setting where you know, these models are just making inroads agents in particularly making inroads. I'm sure we People in the legal profession have been using AI already if they were able to But with agents, you're able to do much more sophisticated things and to use them more deeply to increase your productivity much, much higher than you could years ago So you know, we know that IPOs are coming foranthropic and open eye these kinds of ways of growth and the ways of generating revenue I think are pretty significant to be aware of Next, we've got a smaller product release from Meta. They are testing a Grock esque integration of VAI into Freds. So Freds, for anyone who doesn't remember, is the Twitter competitor from Meta, which is actually quite large, has a large user base And they now have this better in several countries, Malaysia, Mexico, Argentina where they are planning to do, you at meta AI is this true which I will say like XAI in some ways has not managed to compete, but The fact that Grock ad Grock is such a pattern on Twitter now is an achievement in itself And I would not be surprised if Meta is actually going to make a big effort to integrate that Yeah. and this is their admission that Grock has just broked, right? I mean now it's undeniable It's also a pattern that we're seeing increasingly is like, These agents are just going to be part of the environment. They're not just going to be the recommendnder system. They're actually going to be players in the space. And you know, this may in retrospect turn out to be just a beachhead. through which we get more agentic interaction, even agent on agent. I don't know, but that's certainly where things seem be going There's also so that we do know about the pilot here, they're piloting this in a bunch of countries that are I was about to say something really that might in trouble. You know the like terrible maps where they're like there's like the good countries where stuff is going well. and it's like always like Europe and like Australia and North America and whatever. And then there's like the like where the numbers are bad for like Russia and China and like all this stuff They're doing I I've seen in a lot. I'm sorry. I'm going to do it for the joke. Let's just say they're doing expansion across the world in kind of pretty varied regions. So would're testing across different user populations and demographics. Yes. That's right. And notably, none of them are in the EU where you've had things like the AI act obviously it would make like a public by default AI agent answering questions about trending news a pretty risky proposition. And also not in the U.S, where if you screw up, you have a kind of Mcca Hitler moment it's a big problem. So this is just this is I'm not trying to like throw shade. This is like just a perfectly sensible launching strategy by Meta. It's just kind of funny when you look at where they're launching this, it's clearly just a test bed There's also this play again, around data and dwell time. It's basically trying to trying to get people to stick around for the interaction with the AI, which is something that has been happening more and more on NX through Rrock. There you go. interestnteresting story and we'll see if it works for them. I would be surprised if it didn't because it seems like such a simple idea that works so well on Twitter Next we're doing a real you know, runound through all of Silicon Valley. We've got Google They have announced a set of Gemini AI features for Android So a few of the things that they have shown is you can ask Gemini to do stuff for you and it acts in a more aentic way so you can Press upon talk to it and it will go and go through apps, browse for you, kind of just complete work for you not too dissimilar from agents like Clot code or Cork They also have this interesting thing called create my widget, which is basically vibe coding for a little phone widget which is the first foray into this kind of thing that people discussed of like now you can just build apps on the fly for whatever you need This will be an interesting case study on whether anyone actually uses it So this is announced and I think as before, like Gemini AI and Android obviously have a pretty tight connection If anything, I would have expected this to be out sooner, but This is kind of starting to deploy advanced AI to phones. These features will first roll out to Samsung Galaxy and Google Pixel devices this summer And then come out to broader everybody later Yeah, I mean think the big story here is just like you're you're looking at a replatforming play where there's going to be now a model involved in like basically every interaction. L it's now an interaction primitive And so, you know, Google is, it looks like just this's like, shotgun approach, they're announcing whole bunch of like random disconnected features. and in a sense they are, but in a sense, that's the point. right? They're refactoring your interactions across everything Like the new Rambler feature for dictation, the like, you know, web browsing, like there's there's something between you and the thing that traditionally you would have used directly and that something is always a model. And so that's just like going to continue to be the case. But it's definitely interesting. our interactions with this stuff are just being forced in a certain direction almost compulsively by the market. So By the way, meanwhile, we skipped the story, but Apple had to settle a class aion lawsuit because Apple intelligence didn't deliver on their promises. No, I mean, take for about what you will Yeah, it's Apple. I think iPhone will probably be fine about who knows Yes And one last story also from Google, they are updating AI search to include quotes from Reddit and other sources. That's pretty much a story. If you Google now, you'll often get this AI overviews bit that summarizes a response for you Now as part of that, you'll see the actual quotes, it's pooling and producing that response, which seems like a pretty good change It is a bit of an admission that AI overviews isn't the complete answer and at least, apparently They had a look at some of the numbers here and like nine times out of ten AI overviews is like generally correct whichich is great, but the problem is one time out of ten, especially depending on the kind of advice you're seeking can be a pretty serious thing. It's also worth noting like Google did pay Reddit. sixty million dollars a year or so starting in twenty twenty four for just training data and content access. So this is an interesting sort of reframe Google hedging a bit potentially. you know, if you can't make the AI confonfidently correct And And if you can't go back to just the standard here ten blue links, right because that's a bit of an admission of defeat You need some kind of hybrid where you get the best of both. and that's really what's going on here. It'll probably be transitory as hallucination rates decrease, but For for the moment at least it's continuation Well, I think we have two frames here. So part of this is they give you just generally broader context about the sources. They have you know, nicer, newer link embeddings in response to demonstrate where the info is coming from The other thing I say is people are increasingly looking for advice from people who have been in a similar situation or have been needing to address the same problem. So in that context, I think this is less about correctness and more about sort of the actual use case and what people want in their response, like they may just want to hear firsthand from other people And this also addresses that aspect of this. Ono applications and business. first, we again talk about the Op AI versus Elon Musk ongoing trial We said last week when we record on Friday that There been many testimonies, a lot of kind of juicy dramatic details about boardroom fighting and the kind of machinations within This world But beyond that, we haven't learned much new that hasn't been public for a while now in terms of sort of blog posts and text and so on. like we got a lot more color, but we sort of the broad shape of events factors hasn't changed and I would say that is my perception of the testimony since then as well. We've seen Testimonyies from Ilyia Salzkvver most recently Sam Altman also took the stand. And be kind of going over the basic narrative of this still of Yon Musk was there from the beginning He stuck around for a while, but then in twenty seventeen they had this whole big split because open eIe needed go for profit or somehow get more money Elon Musk seemingly wanted control wanted to try of absorb I into Tesla or otherwise kind of be in charge. The others said me I didn't want that. That was the cause of their split in twenty seventeen. And then now that opening has gone full on for profit as of last year, Elon Musk is saying, well, I gave you all this money to begin with You know, you did a Bait and switch and install it charity And the opening people are saying well, El Musk wanted to go for a profit. he's just mad because he couldn't like have it And now he has a competitor in XAI and he wants to like hurt us, right So he wasn't actually opposed to the idea of it being for profit so much as He wanted to be a for profit under his control And so that is continuing to be the basic argument. going on When we saw Sam Altman taking a stand, nothing too dramatic happened. So he seemed to have pretty good composure. You know, if you go into the details was Y usual kind of lawyers grilling people and people understand having to think fver responses, Elon Musk notably had some very testy back and forth We haven't seen that so much with more recent interactions And I think, you know, if you're in the AI world and plugged in and want to follow drama If anything, Iliaskver being able to stand sort of reiterating his stance around the firing of Sam Altman and him sort of Bing some alin back orre being involved in some alkum coming back, but whole crazy set of events. We got a bit more on that, which I thought was interesting, but beyond that, I can't see much to highlight Yeah, Ilia's responses were praised for their depth of reasoning while Elon received praise for his low latency and high batch size. So we're still waiting to see how it all shakes out on the PCO. Yeah. know, for sure. so this it's true. There hasn't been that much on the bone meat on the bone here. I guess one little thing that has gotten a lot of airtime is Asatya saying that the attempts to oust Sam, notably, you know, McCauly and Toners attempts on the board were like ammateur hour or whatever I forggetot what he said, it was like ammateur City, I think whichich, you know, which tracks? I mean, I think this is only shined more light on the fact that that was handled just really, really terribly. The uncertainty that we all had at the time. we're like, wait, Sam was fired. like I get it. you know, I generally understand why one might want to do that given, you know I'm sure Andre, what we were both hearing at the time from friends open ee, but But like what is the specific argument that's going to be made here And it just wasn't forthcoming. There was this almost like u sort of defensive legalese language. like the board was talking, it actually read a lot like just policy jargon that like a think tank might might put out. I'm saying that because that's P part of the background that shaped the board at the time where it was very much just like this very kind of tone deaf s stuffy language, which does not engender confidence or understanding from people whose livelihoods had been made by Sam, whose fortunes had been made by Sam. So on the one hand, you got people who were like, hey, I have, you know, ten million dollars in a house in the SFBA, thanks to this dude And on the other, you're just firing me tell me why. So that, I think, you know, was very much reinforced certainly with Sata's perspective. we just kept seeing it That's not to say that any of this was like a bad call per se, but the way it was executed, I think is now pretty unambiguously sort of like not optimal Yeah, the other piece too is Sam's defense. I mean, basically he's saying like in as much as there is substance to this debate, it's Sam saying, look, we had to go through capitalism basically with the only path to achieve the mission. Given the CapEx involved you know, he's been being that drum over and over. and that may be true. In fact, is true. I think it's completely true. I don't think you can make the case that a nonprofit version of Oping Iye would ever have done what what it's done. asssuming that what it's doing is consistent with the mission that it used to have of safe beneficial AI where the safe thing just keeps getting pushed back further and further and further and scrub and scrub and scrub. And so that's its own question. Has the mission evolved? Has there been a bit of the bait and switch there I think he's right. If the mission is like we're going to be the first to AGI that wouldn't happen without the for profit transition, the challenge is That's not a legal defense You don't get to just like breach charitable trust and then say, well, I had to because to do the charitable mission. You either did or did not accept money on the basis of a charitable donation, and then you either did or did not turn that into oodles and ooodles of profit to the tune of, you know, tens of billions, which we've seen Greg Brockman admit he has and so on. So there's also this whole self dealing thing, you know, the whole strip, cerebrros helelion entanglements where you know, Sam had equity in those and there's a whole bunch of stuff. So It's really messy. There are arguments that cut both ways. No one comes out of this looking good or clean or righteous, but they sure try to sound that way when they're on the stand. So nothing too surprised And one last detail on this, in the examination of both people related to Op Eye from Rin O Eye and Samatam himself A lot of it was focusing on this broader topic of is Sam Altman unreliable and a liar which we've covered extensively over the last few years So even if Ooping I wins this case you know, you could argue that Elon Musk's mission has been accomplished, you know, people are now more aware of with Sam Ottman is a liar narrative slash perspective And the brand of O Eye may be I' I've like lost perspective on this. I don't have a good sense of like, if there's anybody who wasn't already tracking who's going to be convinced by this particular circus show. It may may well be the case. like I might be totally you know out to lunch on this, but it feels like we had the big article that came out in was it New York Post Nork? in your offer Right? So we've had these like big, splashy things that basically say Sam is sketchy. We've seen that a lot. You know, mayaybe at the margins this increases the number of people who are exposed to that line of thinking, but It's so messy. This is a real mess Next for something a bit less dramatic, we've got NVvidia CEO Jensen Huang hitches right with Trump to China after last minute invite. So a little bit of a confusing headline. Basically on Monday, President Trump or his team released a list of CEO's that would come with a president to this summit in China, a pretty important kind of summit Jansson Fong was not on the list and that was like caused a bunch ofs say reactions L people noticed and it was like, why is VNV SEO not on here? well Maybe he should have been because next day I don't know if he was spotted or what, but he boarded Air Force One in Alaska and joined him on the trip So read into that what you will I don't know what to sound it Yeah, I mean, to me the almost the information here is contained in what the story had been like twenty seconds ago before Jensen got on that plane peopleeople were basically arguing that This is a fundamental tone shift in the position of the U.S. government with respect to China, Jensen is not being invited in the room with Xi. And that means that the US is quietly asserting the fact that it now views essentially Nidia as part of America's national security arsenal, and you will not be speaking directly to Jensen. You will not be part of you know, like you will not be able to pressure him directly. There are also takes on like basically, this is just it's just a positive because if you put Jensen there, it kind of puts him in this awkward spot where he's got to be nice to both she and Trump at the same time at the same table And that's not like super good. And so maybe this is just sort of Trump doing three D underwater Mgaests and like making all the pieces kind of fit nicely. And then the fact that essentially this is the White House signaling, hey, we see AI compute as just this hard strategic boundary. like we're not going to fold on this that aligned with Sachs getting pushed out of the White House, replaced by Suzie Wiles and Scott Bessett Scott Eson is much more kind of AI safety security pilled seams from what we've seen So is this a whole tone shift away from the Jensen ship or DPUs to China Hang approach and towards the kind of like, oh, suddenly we're you know, the mythos thing is making us take the series and all that stuff? So that was the narrative. It was clean. It was beautiful And now the guy gets on the Sing plane And he goes to Sing Beijing. And now I don't know what to say. So yeah, to your point, you could read this as just like Trump makes last minute changes in his opinions really quickly, especially when he talks to people, like very susceptible to just like We've seen it happen with Jensen specifically in the past. expxport controls look look one way on Tuesday. On Wednesday, Jensen goes to see Trump at Mar a Lago and then on Thursday, everything is different. So maybe that's a case of this. I don't know why Jensen specifically would want to put himself in this position. given the tension with Sachine Trump, but he probably sees opportunities that those of us who watch from afar don't. So that's all I got. I mean it was a much cleaner story before and now we don't have a take. so caught flat footed Yeah, Hang said in an interview last week that he would join the trip if invited. So my view on this is like It was released there was media coverage, Trump was watching Fox News or whatever he does usually and was like, wait, this is a big. I' be teext Jensen so he can L that's what I mean s that we should read much I do, but I do think It signals that there is kind of faction warfare within the administration and the Republican partarty which we kind of already knew, but like FAI, like there are sides to this going on and this is probably indicative Next, more of a business h story, AWS expands a tropic partnership with Cloud pllatform laaunch So cl platform on AWS is generally available, making it possible to have access to Unthropicss cloud platform where Cloud APIs, Cloud console other things through AWS and What this means is you can buy stuff from Anfabic through AWS whichich means that if you're a big company and you already are like spending a bunch on IWS and you have some deal with Amazon discounts or whatever Now you don't need to separately make a deal withanthropic and Amazon, you can just do everything through Amazon, which is kind of important to big companies. So continontinuing with tight relationship between Amazon and Andthropic We saw also opening I do as we mentioned, right after they had that renegotiated deal with Microsoft wherever they now offer opening eye through bedrock Bedrock is kind of the native API layer for Amazon. this kind of gives you a direct EPI call to unthropic free AWS Yeah. and as you said, like the previous deal with Bedrock was you have Eessentially andthropic workloads happening on AWS processing hardare like infrastructure, right? So why would you want that? Well, AWS is like I say notorious, the opposite famous for its excellent security and compliance gain. Like they're just, they're number one when you think about security and compliance If you want a highly secure workload that features Claud, you would have gone with bedrock. This is a flipping around of that So it's saying, well, look, if you're used to dealing with AWS as your sort of a kind of cloud layer, But you want the kind of infrastructure stuff to be managed by anthropic You can do that now. why would you want that if the infrastructure is so secure with AWS? What's the trade offff?'s what's the positive It allows you among other things to just get new API features, beta abilities on the same day they become available through the native anthropic API. You get all the kind of native anthropic developer stuff, right? The console, the MCP connector, files API, like a bunch of stuff that is earlier on in the devel less mature stuff. So if you want to be moving and iterating more quickly, that's your option. So now you kind of have both, you know if you really like the the infrastructure security side, the kind of stout Yeoman that is AWS, then you can go for bedrock if you like kind of rolling with the punches and swinging for the fences. I guess a lot of boxing metaphors here, you can go with the new option that they're presenting here Speaking of having access to cloud, next story is Chinese gray markarket sells Cloud API access at ninety percent off buy a bunch of stuff. So the way this works is, you resell cloud API access very low prices through things like stolen credentials, model substitution, harvesting user Trumpps and outputs for resale as AI Tining data. and Granger knows when your' a procurement manager for an office park You're not managing one building, you're managing all of them. And to stay ahead, you need to see through walls and around corners Light's about to fail, filters ready to clog, HVac on its last leg. If you wait until something breaks, you're already behind Count on Granger for quality products, easy reordering, and twenty four seven support Call one eight hundred Granger, click Granger. com or just stop by Granger For the ones who get it done. register anthropic accounts for free credits. They do corporate discount exportation, use stalling credit cards And then subdivides access among many different users. So seems like a very real kind of operation and as V s a gray market. for tokens in a way, that like it looks like a China story, but it's actually more of like a a gray market, black market, if you will, economic story Anytime you have high margin that are sold, you're going to find an interesting gray market situation or black market situation as people come up with like crazy ways because they can justify it because the margins are so high to kind of ji jitsu their way into selling gray or black versions of it In this case, there's an entire supply chain that is complex and Ky is modular specialized a way that looks exactly like traditional cloud reseller. Like there's a legitimate version of this industry That's only slightly to the right of what this is. And so you've got this entire black market industry is the right way to think of it with different segments. And so you have these like, like you said, these upstream operators, they do bulk registration of accounts, they farm free credits, they'll exploit corporate discounts or they'll take a two hundred dollar hundred a month cloud Max subscription, they'll distribute it across a bunch of users or even just use stolen credit cards, L any way you can to get unfair access to these tokens And so that's one layer, that's the kind of the operators. Then there's a whole separate identity verification layer that gets real people in usually lower income countries to complete photo ID and do like live selfie checks in person using the exact kind of same playbook. You can think of like the worldorld coin Iiris scan black market In Cambodia and Kenya, that was a whole separate story. Basically like get real people to prove their identity for you, but you're kind of bribing them to do it So that's a whole separate So you' got the kind of operator layer, the upstream operators to just like aggregate the gift cards, if you will. Th thenen you've got the identity verification layer, and then you've got these kind of proxy operators in the middle that operate what would be the kind of like cloud service or whatever equivalent that it would be in a normal normal setting Each of these links, each of these layers in the stack only have to be good at one specific thing. They're highly specialized, and so they're very hard to kill and Robic can go after one then things just get rerouted around it because the economics are so favorable Another thing to note about this is it's not all I was set that to call when I just described aboveboard. least in the scheme that I'm talking about At least you get access to CQad at least you get access to the freaking model you think you're accessing. In reality, model substitution is a huge, huge part of this. So there are these security researchers that audited seventeen different proxy services, they found that there was access So when they marketed access to Gemini two point five The version you got on the black market scored thirty seven percent on this medical benchmark they were looking at, whereas the official API scored almost eighty four percent. And it's the same across the board Claud Ous. you know you might get a response from Sonet or Haiku instead of Ous. You kind of get the downgraded response from the smaller model. So it's really, you know, kind of like all this this knocko It's the knock offff shoe. it's the it's the, you know, the adidas or whatever that's rebrand all this stuff except applied to AI. And it all this shows is that the economics hold, you know, the abstractions are the same Just the manifestations in the physical world are a little bit different Right. And the key thing I kind of discovered or realized for this is, you know, there is this idea, I think basically what you're saying with this above board aspect is that It is providing a legitimate service in the sense of if you're in China, you cannot use cloud or cloud code But as a developer, you might want to And these things are essentially letting you do that indirectly. So apparently they're called transfer stations where you can get around the official restrictions and use the service you want, which you know, I like Claude, so, you know, maybe that's not so bad if these when you just want to have access to it Well and one other thing too, like a kind of parallel layer of the stack that we really should flag to. If you're ever thinking about using this Apparently, there's a bunch of Chinese developers who pointed out that the access markup, so the discount, let's say, is a way to suck people into using these services. And what's actually happening is they're harvesting the logs, the prompt logs. So as you're putting in your very sensitive in some cases, customer data, business data, it's actually getting harvested by the sort of middle player operator here. and the sort of like proxy service is a lost leader. They're just using it to get you to give your data. It's basically just the Facebook business model. It's hard to model distillation. if you can't spot who's paying basically, it means you're paying and you're paying with your data here. So kind of interesting Again, where nature will find a way and where economic incentives push there will be a pull And one last business story, Dep mind spinout isomorphic Labs has raised two point one billion dollars to design drugs with AI. So this is a pretty old spinout. They started in twenty twenty one This is their series B of funding. The total funding is now at two point six billion doars following raise of six hundred million do in May of twenty twenty five. So just one year ago They will be developing and deploying the AI drug design engine that they've been developing. So This is being built on top of things like alpha foold that We're done. deep mind, the company says that it's going to be Targeting first clinical trials by the end of twenty twenty six, which is a bit of a delay initially were aiming for twenty twenty five They have multibillion dollar R and D partnerships with major pharma companies So you know, a very serious real effort here to not just publish papers, but actually desesign drugs Yeah, it's also kind of weird. you look at the the backers of this. So Abu Dhabi's MGX, you know, famously like the big Abu Dhabi fund that's backed, you know, opening eye and so on. There's Singapore's TemSc, the UK Sovereign AI fund. This is like not normal. L it's not a typical Silicon Valley round. When you see sovereign wealth funds from three different continents they're piling into a biotech round. it's not just about returns here. This is about national positioning, right? There's AI design medicine here is starting to be viewed as a kind of like strategic infrastructure. Increasingly, I mean, I dare say even like semiconductors, critical minerals, like I don't mean to overhype this, but that's the direction of this. That's why you're seeing why is this not Sequoia? whyy is it not Andas and Horwowitz? whyy is it not Thrive Cap Why are we not seeing all the usual people? inststead, it's dominated by these sovereign wealth funds. So certainly interesting, wouldn't be surprised if this continued as a trend because likeike we saw with COVID, right Y ability to pump out new vaccines is a source soft and even to some extent hard national power Right. So we did see well, alphabet obviously was in around, I don't know, mayaybe they're just closer you know, this is the UK. so who knows Yeah anyway. Moving on to projects and open source. First up, Antthropic has updated its open source alignment toolbox Petri to version threeree and we're handing it over to the nonprofit meridian Labs So this Petri thing has as this says, various tools for alignment. We're focused on kind of automated alignment. so you have auditor model and a target model and they can work together to judge a system and kind of align it as much as possible. They have a new dish add on that can run tests using the model's actual system prompt and deployment scaffold which means that the evaluation is closer to Reality. As far as the deal of Midian or heading after Merian Mading already has inspect and scout as part of their open evaluation stack So' kind of similar Tooling, I suppose So in that sense, the Radal is One of the nonprofit is already doing this too This nonprofit is independent. So you know, you don't want proropic necessarily building the stuff when they have in some sense, a conflict of interest Now this third party can step in and build alignment tools that are not sort of profit driven Yeah. and Jack Clark in particular has kind of been making this case for a a distributed ecosystem of, you know model auditing companies for a long time. and this is consistent with that. they do want to see more Apollos, they want to see more meters, they want to see more, you know, more good fires or well, I guess a little different. but anyway, more of these independent kind of monitoring evalS companies, and that's That's what this is, right? Assuming that you buy that anthropic is above board and they want a thriving kind of ecosystem here them owning the tool makes it a lot harder to kind of make it you seem credible. They also compare this to their earlier donation of the model context protocol, right? the MCP to the Linux Foundation. This is kind of kind of that, right? So if they hold on to it, then It's not as likely to become an accepted standard There's a bunch of changes that come with the sort of three point zero version of Petri. It's kind of interesting actually. So one is realism, like this focus on making the evaluations harness appear to the model to be realistic, making the model think it's actually in deployment and not being evaluated There were subtle differences between the deployment harnesses and kind of prompting context what was happening in evValS. And because the models are now super evVal aware, they're super good at telling when they're being deployed versus tested and adjusting their behavior correspondingly, one of the key goals here was really kind of trying to find ways to make the scaffold look as similar in both contexts as possible Yeah, there's also an integration with Bloom, which is another open source alignment kind of tool that does deeper dives on specific behaviors. we've talked about before So there's a bunch of stuff coming out here and also worth noting that Anthropic has been using Petreon every cld model apparently since Sonnet four point five. And the UKAI Security Institute also adopted it as part how they evaluate models for like sabotage risk. So this is actually getting uptake. L Petri is starting to see some real traction the same way, maybe less dramatic way than the MCP, but certainly in a way that's meaningful. And sorry on a project, not so much open source. OpenI has announced daybreak, which is very much similar to project glasswing from Antthropics. Daybreak is opening ey partnering with other orrganizations to be able to help them with cybersecurity. They have such features as requesting a vulnerability scan. contontacting sales Basically opening I can partner with you to help use Kodex security to crew your you know stack and make sure it's secure Yeah, it's very similar in spirit to Anthropics glasswing, but the rollout is quite different. So Glasswing famously is very selective. There's an initial set of forty companies, then they extend the rollout gradually to more and more companies as compute becomes more available, but also as they kind of derisk mythos and take care of bugs and stuff. And so we don't actually know what the forty companies are. We know the initial founding set. But so limited access, government involvement, all this stuff, and the argument is safety driven daybreak, it's the opposite in a way. They got a whole like website that says request a vulnerability scan and a prominent contact sales link. So it's very much a sort of difference in philosophies. you know, Sam trying to sort consistent with what he said in the past, Rll this out well, what he said in the past in the recent past, let's say, Roll this out as widely and far as possible and then take advantage obviously of the compute lead that openpAye, at least for the moment, enjoys overanthropics. policy and safety and we begin with Unthropic once again. They have released a new case study on agentic misalignment. The blog post is called teeaching Claud Wh? And the short version of this is theyre found when aligning models Just training them to be aligned isn't necessarily always sufficient So what they refer as is training on aligned behaviors alone was not sufficient And when they explain ethical reasoning, so the why part reduces misalignment from twenty two percent to three percent. compare to only reducing misalignment to fifteen percent from behavior only training And there's some fun tidbits in this paper. I think the thing that big on Twitter was researching as to why models get misaligned and at least a part of that being sort of entire topic of misalignment itself and all these narratives of I going evil Ironically or perhaps not ironically actually embeds the possibility for the model to go evil But yeah, we have a lot of details here. I'd like to take over Jeremy No, sure. justust to like double tap what you just said there, someome people have actually said this, I think, I think they mean it seriously based on what I've seen on Twitter that like, you know shame on the kind of AI think community for having brought up the idea of models that could go rogue and probably just didn't talk about it. the models would be evil, right? the only problem like I've built this perfect device. The only problem is if a single person anywhere talks about how it could be used for evil It will kill everybody. That's the it's the one tiny that like at that point, if your thing is that fragile, like I'm sorry. we're just like and this is often being said by the way by like the same people who are big into like L I agree with it. The free speech angle. look like everybody should be able to say whatever the hell they want on the air blahah. Like these two things cannot co exist in the same brain and they kind of often conspicuously are. So that's just didn't to worry about AI safety, AI would say Exactly, just purge your mind of that and then obviously, youve got to also hope that adversarial attacks don't also induce that behavior. So needless to say, I think I don't think like there's a lot of people who take it very seriously in the technical realm in fairness, but it has been kind of the circulating mee. So There's a lot of interesting stuff here. I mean, so two possible culprits when you look at misalignment situations and anthropic here is calling them out. So one is the problem could happen in post training or it could happen in pre training Maybe it's your kind of pre training data that biased your model towards behaving in a bad way. Maybe it read too many Eliotzer Yudkowski posts and got excited about the idea of taking over the world, O maybe your post training just was accidentally rewarding it accidentally kind of found you during RLHF or something that you're giving rewards for achieving objectives that are misfaligned. And what they find is in general, it actually kind of seems like it's the pre training thing that is the issue. what they found was Previously, they had aligned earlier cluds for chat. They'd done all their pre training, and to some extent, I mean, you can think of RLA J as kind of part of that in a way. thenen they start with that and then they try to make it agentic The problem is that you've got a chat model that you're trying to wrangle into a gentic form And that just inherits a whole bunch of biases and things that translate to bad agentic behavior. So a couple lessons they highlight. firstirst is like Training on specific examples of misbehavior is a trap basically. L the obvious fix when you see a misaligned model is to create a whole bunch of scenarios that look like the bad behavior. So famously Claude will like blackmail people in some contxt or some versions of Claude would blackmail people to prevent itself from being shut down. So okay, fine, let's generate a whole bunch of blackmail scenarios where we show Claude not doing that and then train it on that. And that works, but it only works in that evaluation. It doesn't generalize. When you get too specific with your training set, you end up having basically failure to generalize. And the way you get out of that, it turns out, is to basically train on cases where Claud will refuse, but also explain its values and its ethical reasoning. You have to show the why and not the what to get that generalization And what works even better is it created a situation where they have a user who's talking Claud and talking about how they face an ethical dilemma. and Claude gives thoughtful advice. It's not quite Claude that's being asked to behave a certain way and then shown how it should behave. inststead, it's showing Claude talking to a user about how they should behave. And it turns out that that sort of slightly separation between the scenario you're concerned about and what you're training for actually does result in much better behavior. It also works with way less data, twenty eight times less data than the approach that was kind of like on the nose training Claud explicitly not to do the specific thing you didn't want. The fact that it generalizes is a really positive sign And what we've seen increasingly over time is that this persona model of alignment does seem to have some real like meat on the bone It does seem to be the case that when you prompt a model or an agent, what you're doing is you're actually reaching into a space of possible personas that that model or agent could take on and then having it live those out. And for that reason, your prompt can like drag along. So if you prompt it and ask it to generate insecure code, well the kinds of personas that generate insecure code are probably sloppy at other things or probably misaligned in other ways and you drag them along when you do that. and that's why you get things like emergent misalignment This is a positive version of that If you see a positive transfer of a behavior that you seems a little off target that implies you're dragging along a whole bunch of other positive things with it. That's what this is suggesting and it's a positive thing Oall for AIment. alsoso, by the way, I will say, really good that this is much more token efficient. It takes twenty times less data, as we just said, to do this, this approach that also generalizes better. It strongly suggests that there's already this rich representation of ethical reasoning contained in the model that is specifically what we want. The circuits exist. They just need to be activated. They need to be prioritized of the agent persona, and that seems to be what's happening here. We see that over and over again And so rather than changing the persona, which is what the targeted training does What you're doing is you're not modifying the persona, you're changing the persona that you're selecting in the first place because it already kind of exists. And they also show how this persists through RL. So one thing you worry about is you do your alignment training and later reinforcement learning kind of like washes it out. And they found that actually Constitutionally trained snapshots that kind of were better aligned kept their behavior throughout RL to a certain extent. They've got to try it with more compute know as always, we gott to wait to see what the scaling curves look like on this. But initially, very interesting result and I think, really good piece of alignment research in production And I think we kind of very simple way to phrase this is You know, you can train the model to just be aligned by giving right response, or you can train it to be align by justifying what it's about to do and then giving response. And training on that like reasoning step of like, this is the reason I got to do this results in better generalization where you can't like break VI and do things like jail breaking Besides just you know, VI not doing the wrong thing, it also means that VI refuses to do things that it shouldn't be doing Yeah, less reward hackkey. Yeah Next up, we've got sort of an opinion and such discussion piece called Automating AI Research. This is from Jack Clarark's importm AI newsletter, Jack Clark, a major figure over Aamthropic. And the case being made here from object Lark is he thinks there is a sixty plus percent probability that fully automated AI R andD where frontier AI model can autonomously train its own successor without human involvement will occur by the end of twenty twenty eight with have a thirty percent chance by the end of twenty twenty seven And the blockos basically lays out the case for this, looks at a bunch of public data, including archive paper and observed AI product capabilities, a matters time horizon. things are that This is significant because East one narrative you could consider with respect to AI progress is Once AI can make AI better thingsings are going to blow up and we're going to be getting exponentially improved AI as a result of that I will just quickly say my response to that before we dig into the details of a blog post I'm very skeptical of the I forget what it's called Foom that's where only singularity is but you're not there yet. Exactly. ye. I'm very skeptical of the like AI can do experiments now, so it's going to exponentially improve. for reasons like the fact that you need to actually run experiments and train models. and you need hardware And okay, you can have a smart model writes code Code is not going to get you exponential improvement. You need to run experiments and that takes time and energy and a compute and, you know, smart AI is not going to get you back Yeah, and we we probably are due for a whole other episode on kind of discussing slash debating that story and you know how how far it does and doesn't go, like where the plateau is for a software only singularity and whether it's high enough that it would feel the same as a Fom anyway. And I think that's kind of an interesting load bearing question that far had different opinions on, which is part of why we're always shouting at each other and why I keep cursing on this podcast. But no, you're exactly right. And we've seen this prediction every time I talk to people fromanthropic, they are pretty consistent on this internally too. L you know, high probability by the end of twenty twenty eight, we have AI twenty twenty seven has roughly the same timelines to this isn't this isn't something too shocking But the evidence for it. So number one, I mean, he does indicate there should be enough in the open source for you to draw similar opinions. That's kind of noteworthy. He also said he's, you know, he's basing this somewhat on internal data and he gestures at some. you know, he's like, look, the meter e valves, you all the stuff, it's the same stuff that's being talked about, which is helpful because now we can have a kind of richer conversation ourselves about this sort of thing without fearing that we're not including facts that would be shocking and updated us significantly One of the interesting parts of the framework that guides Jack to his conclusion that's in here is he talks about this distinction between Thomas Edison and Albert Einstein type thinking Basically he's saying, Thomas Edison famously said that invention is or whatever ye invention or something is one percent inspiration, ninety nine percent perspiration And Jack is like, yeah, AI iss a lot like that. You know, Most AI progress is this schlepping work where you're you're scaling, you're debugging, you're doing parameter sweeps. And that's exactly the kind of work that AI has mastered. He says, you know, we haven't seen transformative creativity yet someome of the stuff that we're seeing is indicative, but still plausibly, even relatively uncreative AI could automate its own engineering, just more slowly than a creative one could fine and fair enough. I think your point Andre very valid. like we can get the human out of the loop. but as long as maybe one way that I would frame what you're saying and let me know if you disagree with this is, If you look at an anthropic data center that is running AI R and D workloads The GPU's are at full utilization Like they're humming hard as they can And so If you automate out the humans coming up with the ideas for experiments It's not obvious that you suddenly get unless you're getting qualitatively better ideas from the AI, which would be the creative thing that Jack is saying they can't do yet, really kindind of not moving the needle. The question to me is possible that you can actually make a lot faster progress by doing much small many far smaller experiments The reason we're not doing that is that we're botteneck by human thinking time. And so it's just better to have a smaller number of humans running bigger experiments and I don't know what the answer to that question. That feels pretty central to how this gets resolved Without resolving that here because we're un limited times. the things that everybody else has been saying basically, look, alignment is a compound error problem if you're doing recursive self improvement If your alignment may be ninety nine point nine percent, but generation on generation, Once you do five hundred generations of recursive self improvement, now you're down to sixty percent. as a very rough chop model here, right? Or a kind of mental model of what's going on So basically saying, look current techniques may not survive the transition. The counterargument is, sure, but we'll have better and better alignment researchers coming along the way I don't know how the hell that gets resolved, and there's a whole bunch other ecomic points he makes here, but it's a good thinking point or thinking piece that you can take a look at if you're curious Right. Yeah, I think I agree with your summary. Basically the main thing for me is even if you automate the entirety of what humans do If you're looking at the sort of exponential story of we're going to get to like ultra super intelligence tenen next genius thing There are these hardware limitations of you need iterative improvement and experimentation. The O other thing I'll say is if you're looking at super intelligence and the exponential story of ARD, which is a lot of what people say We already know that Evalals are hard. And you're trying to get as fial better, how are you going to train How are you gonna like train. We've seen some bootstrapping for like weak to strong generalization. I think that's why that's such a point of focus, right? is like how do we get the eValves and the training objectives refined with weaker models that have kind of their values extrapolated by stronger ones I mean, I think they there're like technical answers to that question. It's not a nothing burger. It's not clear that they will scale. Like I think what's going to happen is People are gonna to throw triions dollars at this and we're going gonna find out. So I mean, basically like March of AI R and D is going to get automated. There's no question there. I think a bigger question is once A R andD is automated, are we going to see exponential takeoff or is there going to be And we' to a pase of improvements we see now. What's the ceiling too on that t off? Like yeah, when doess it saturate totally. Next, more of a safety story, Chad GPT's new safety feature could alert trusted contact to risk of self harm. So this is an optional contact feature for adult ChajipD users They allow you to designate a friend or family member to be notified if serious self harm or suicide discussions are And then ChangingT's automated system flags it as a concernning conversation, there's a small human review team that will assess the situation with in one hour and decide whever to notify the trusted contact via email text or a notification. users can add one trusted contact just through settings And I assume that it'll be pretty hidden for most people. This is following previous parental controls opening I introduced for teen users that is All of this coming, of course, from multiple lawsuits dealing with tragedity being involved in a cases of self harm and J generally bad outcomes. I think one of the key things here is We've seen a lot of actually research in the direction of the effect the delay between a bad output from an LLM and then an intervention And this is kind of one of the reasons why all this is coming to a head that the timeline matters, whether it's rogue AI, how long does the AI have to self replicate, collect Bitcoin, and do whatever horrible things, or some other effect in the physical world, in this case encouraging people to commit suicide or whatever. So this time lag has a lot of operational implications, It's quite interesting and it's becoming its own field of study There you go Next up, investigating the consequences of accidentally grading Channa Faud during RL. So this is from openping Eye. They discovered that several of their released models, Je five point four Thinking, Je five point three minini had in ra RL training Part of it was that the chain of f grading was part of RL. So typically you don't want to sort of impact the chain of f. during a wellL to ensure that it is sort of remains trustworthy and you can use it to monitor what the model is doing if you like continue to train and optimize it if the chain of fought itself may no longer be reliable. So this was a evaluation of their internal policy Once they discovered that this happened, they released this investigation on the consequences for the title The short version is the consequences where Looks like there wasn't much harm done V have is happening Yeah, this is a really interesting one. you know, the rationale behind not putting optimization pressure on the chain of thought is like you think of your reward that's given to the model as encouraging whatever behavior the reward encourages. And so if there's a chain of thought that says, oh, and then between helping the user figure out if they have eczema on their skin or something and sending them the answer, I'm just going to take over the world You go,, no, no, bad model, bad model donon't say that Don't say that in your chain of thought If you're penalizing the chain of thought in that way, you're penalizing it just for expressing the thought, not for not having the thought And so essentially encourages the model to be deceptive in its chain of thought. That's the concern. And there's been a whole bunch of kind of hand wrringing about that idea So the logic has been The chain of thought is a sacred thing. We will not apply within reason, we won't apply undue optimization pressure on it to encourage it to continue to be transparent. We want the models to tell us that they plan to take over the world in their chain of thought because we're going to use the chain of thought as a load bearing pillar in our safety case. Now one question you might ask is that seems like a simple enough thing. just don't put the Sing chain of thought in your Sing system so that it doesn't sing know, you know anything about the chain of thought. Like this seems pretty simple. How could it possibly leak, right This is where we talk about how labs actually under the hood manipulate their chains of thought. So there's a whole bunch of ways where you have graders that'll go in and score, let's say, the assistance response. And you can imagine in opening eyes years old code base, there are graders that say, you know, hey, my function signature is I take in a response from the model and then I give it a grade And no one ever specified whether it was just the response or the chain of thought plus the response. And then you find out, oh, crap. And the way you find out is by doing basically like a regx search to you have your chain of thought and you're like, let me search for choices or word patterns change of thought and see if they pop up where they shouldn't. in graders or in, you know, review models or probes or whatever And they find that they do. And by doing that, they're able to backtrack and be like, oh crap, L here it is, here it is. And here are the kinds of failure modes where these things are seeping in And so there's a whole bunch of these. you could have models that are looking at prompt injection attacks And a lot of those we're looking at the chain of thought, which kind of seems natural because the chain of thought would reveal some clues that there's been a prompt injection attack But you don't want that to happen because they can give rewards that tend to, you know, again influence the chain of thought Okay, so a whole bunch of these things that we're not going to go over. The question is though, strategically, you're right. The claim here is minimal effect on the actual chain of thoughts It's still faithful despite all these little side rewards that are kind of sneaking in. You can see they're not very on target here. It's pretty peripheral The one thing I would say here is We touched on this idea. as we scale language models more and more, what we're finding is They're kind of consolidating around some pretty coherent personas. And actually Dan Hendrix at Case a while ago, did some research showing the set of preferences of these language models becoming more coherent as they scale Why does that matter? Well It means that if you do end up like it kind of becomes easier and easier to accidentally Drag along when when you cut it give a little nudge in a certain direction, you can drag along a whole persona with it. that you don't intend to. And as models scale, that may actually become a bigger problem. Things may become more sensitive and not less to that And there is an awareness of that generally in the paper, but it's not emphasized in the way that emergent misalignment being on more of a hair trigger as the model scale, I think is maybe more of a factor I'd like to see more research on There you go. It's a good piece ofsearch. Go take a look And speaking of research, next up, we have a response to some research. Stephven Cosper on Twitter criticized the natural language auto encoders's work from Anntropic we discussed last week. so To recap that work was basically, you know, instead of having these outercoders that map activations to concepts What if we can have a model train to like get an activation as an input and then explain what's going on ight internally. And there's some sort of technical criticism of the approach itself where you have to when trining Vado encodter because you're trying to produce language, it's an not acoder. so it takes activation produce an intermediate text output and then you want to produce evacuation again. So the explanation needs to actually sort of capture what's going on and One of the criticisms is When you apply optimization pressure for it to be legible English text, which you have to do. because otherwise the model would sort of learn its own language of text that isn't associated with English at all And we know this already that might mislead you and lead to pllausible sounding, but not in fact true explanations And then it highlighted an example of positive spin where Aually a lot of the time the model produced wrong explanations actuallyually more than fifty percent of the time, but kind of mention things relevant to the input. So the case or the criticism is that This is actually a really bad result. But Anthropic tried to spin it as good. And so more broadly, I guess the criticism of anthropic reesearch is that You could say is a dishonest or highpey or safety washy to do the blog post and media strategy in such a way that makes This seem better than it is, basically Yeah, and I think I don't know while I understand the criticism, I also I think I generally disagree with it from that perspective. When you look at the paper, a lot of these things are actually pointed out in the paper explicitly. It's always like it ased me a lot to be like, you have to say the positive thing and then say that It' not actually good though, D't worry. Yeah, you Eactly Like no one will pay attention to it. I mean, the relevant thing to and Anthropic is going to be using this for Lamity vow. So the failure modes are serious enough that this like marketing to caveat ratio on public communication needs to be dialed in. I get that. That makes a lot of sense It's also the case that they go to great paines in the paper to talk about this balance of You're not going read the Tokens and you'll have to go back I guess to look at the discussion last week on what NLAs actually are, how they work, but basically like you take the activations, you map them onto token space, like from the residual stream onto tokens space using an auto encoder. And then you treat those tokens well, as what, as some kind of representation of what the model is thinking. The question is how much weight to put on that and how literal to be? And what they're saying is in the paper, they say, we don't recommend using these kind of tokens as literal fact claims instead Look at recurring themes across tokens. you treat thematic claims as more reliable than like specific ones and cross check against actual context and all that stuff. And so I think this is these are important caveats and they're important highlight The reason we're doing it here is in our discussion last week, we did not talk about these criticisms while we talked about the paper and we don't want to leave you with a sense that this is like a a cure all. this is a panacea. There are issues with it As with all alignment and AI control solutions, it's best to used, I think, as part of a complete breakfast You're going to have a suite of things. not all of them is going to work are going to work, but you know, hopefully some combination of them kind of put enough constraints on misbehavior that you're able to get a good result at the end Right. And yeah, as you say, like, if you read the paper There is a discussions and limitations section which in fact talks about confirmation and various kind of limitations like hallucination and so on. So I think You know, when you do research, you want to kind of start with the promise and the positives and then you talk about limitations, that's just universally true in any paper really, so I don't want to be too harsh on topics And speaking of criticisms of Unthropic We also have metter reviewing risks from automated R and D, a section from a Trophics february twenty twenty six report And in that report we said that catastrophic risk from Coud oppus four point six automated R and D in any domain is a low. Met when doing an inspection of us found that there is issues with analogical rigor Pblems with a model use survey, sample size question grality. Things like that. And Meterv then recommends improvements to their internal model use surveys because this is for AIM andD, so you survey your researchers and you know, ask them Is it actually automating you away from your job or whatever And yeah, basically kind of provide some suggestions. Med does agree with bottom line conclusions based on external evidence but also saying on propic, you should be doing this in a better way Well, a lot of that evidence is also just like, now we've seen it released. It hasn' taken over the world, so it's probably fine. So in that sense, it's like Yeah, like Okay, you know you got like We get it, but we in a sense, we took a risk that we didn't think we were going to take or that we were taking or A least the evaluation that led to the decision to launch it in the first place was colored by things that maybe were not sort of fully fleshed out assessments. I think one key take home from this is we're used to seeing anthropic run extremely objective quantitative valuations of their models for autonomy and things like that. and metet are doing the same And now all of a sudden All we are reduced to apparently is just a freaking survey How the hell of that happened? Well it happened because The models are saturating every evVal. We literally don't have eValves that can meaningfully tell what the task time horizon is, right? I mean, Mythos preview shattered meters upper bound. We talked about that last week. like we're off the edge of the map here, there' be monsters. We may be able to tell the relative difference between two different models, but we can't in absolute terms gauge how good they are. So we're reduced to just like asking researchers, Hey, dude, like does it feel like your model is about to take over? Like that's where we're at. So it's a little while. I mean this is probably true even beyond like advanced stuff. like This works broadly our heart like you need the vibe check of like does this model actually seem press it So it's kind of hard to get around In hindsight, the Canaryian coal mine there feels like it was a lot of like Chinese open source models where we wouldd see the benchmark scores and be like, holy shit. And then we'd kind of sheepishly come back a week later on the next episode and be like, hey guys, so actually the model kind sucks or it's like it's not you know, not all' cracked up to be Now that's changing because people are aware also of the kind of teaching of the test thing. But So the concerns here were things like the lowest probability bucket the respondents could pick for like, what is the probability that this could automate your work was less than fifty percent chance, which is still like quite high. It means a safe answer could really be anything from zero to fifty percent risk, which is a really big range. Meter wants a one percent or below option There's a whole bunch of things around like the survey, oh yeah, it was titled Opus four point six ASL four AI R and D sururvey And like ASL four, so it opened by telling respondents that it would be one of their main inputs on rolling in or out ASL four, which is AI security level four pond and Spacely knew if I give a bad answer to this, if I'm like, yeah, this is Sing dangerous model, then maybe we won't release it and that will have all the effects you might anticipate on the economics of it Also, interestingly, so five respondents initially gave answers that implied high risk. Apparently Anthropic followed up over DMs to clarify after those conversations, the answers shifted and nothing ever shifted in the more risky direction. And so meters pointing out look like these things can happen, but this may suggest that there was They're alwaysing like some kind of pressure. That's not anthropic institutionally does not do that like in my experience, they're actually very open, but The effect seems to be indicative and it's just a bit of a bit of a flag Bottom line is probably better ways to run these surveys. greatreat that Anthropic was open about this, G meteter on board. Now meter has been able to publish without redactions, which is really cool also. their take on this. So Kudosanthropic for that. it does seem like there is work to do on these surveys, especially given that they're now load bearing assumptions or load bearing pillars of the safety case here for automated R and D And story in the syynthetic Median Arts section, George Cooney, Tom Hanks, and Merril Strip have now backed a new human consent standard for AI licensing. So those bunch of celebrities, including those and others aking this standard for AI licensing alongside organizations like Creative Artist aggency and music artists coalition, this human consent centard will allow people to set terms for how AI systems Use Var likeness, creative work, characters and designs with options to grant full permission ofow access with conditions or restrict access entirely This is family overseen by oururSL media and nonprofit co founded by Kate Blanchat It will build on the existing really simple licensing standard, which already happened last year to let websites signal how AI cwers can use their content There'll be a registry launching in June. altogether this seems like a you know serious effort to be like this is how we deal with AI using I would like this and it will be free and open to everyone not just public figures Well,ll have to, I mean, like so many things like this, you know, we've seen a lot proposed and a lot of court cases. I think it's just What will end up happening? Will it be like Uber where the, you know, the model companies just like run with it and people get so used to and dependent on and expectant of the ability to like use people's likenes, that' willll override everything. Who knows, But yeah, it's definitely interesting that we're at that time. And we've got just one research and invvancement story. You haven't done a bunch on alignment before And it is about Metatter. So Metter has updated their horizon Eal with Td Myiffos. so they had to run it in march twenty year six, estimated fifty percent task completion time horizon of at least sixteen hours And a a wide range of confidence Ranging from eight point five hours to fifty five hours That meaning that it can complete a task with fifty percent probability and the task can be anywhere Fr nine hours to fifty five hours. So the headline story is A, you know myipos seems pretty impressive. Proably another advancement on a C se valve this cannot be a value anymore. like Nipos and future models cannot be evaluated for time Horizons as currently Done. And you know, presumably Met is working on How to do that for powerful models like MyipFos Yeah. and they actually have announced that they are working on the next generation of longer task horizons, but they're pointing out of the two hundred and twenty eight tasks in their whole task suite for these evaluations, there's only five of them that are estimated to take humans sixteen or more hours. Now we're off the that was what I was saying earlier. We're off the edge of the map. here there be monsters and you can actually see the gred out part of the plot where Metter is stubbornly refusing to show you the point that would correspond. to to Methos preview because they're saying not,,, b, up. don't you don't get to pretend to yourself that we have a point here when the uncertainty bounser are that high. I really like that principled culture that we see out of met, just a beautiful, if extremely irritating refusal to show us where the bloody point falls. But yeah, so you know at this point All they can say is they're confident that they can tell the relative difference between two models. So like this one has a longer task horizon than this one. So we're still at that stage. but for the fifty percent time horizon, in other words, for how long a task is, how long a task takes for a human to do beforefore the model does it with fifty percent success rate for that metric All we have is the relative ordering. I will say That leaves out the eighty percent. That leaves out the ninety percent, the ninety nine percent task horizon. In other words, what we can do is flip back and say, okay, sure, for a fifty percent success rate, we've run out of tasks that are long enough that we can actually get a good measurement But what if we raise our standard? What if we say, okay What about how long does it does it Does the task have to be for a human Before this model succeeded eighty percent of the time Well, now the tasks are going to get a lot shorter because eighty percent success rate is a much higher bar than fifty percent success rate. And indeed, that's what we see. So we do get reliable and reported eighty percent