Final Thoughts on Digicert and Microsoft

It's time for security now. Steve Gibson is here. The FCC has backed down a little bit, and that's good news for router manufacturers. AI has found a 21-year-old critical flaw in well, the most secure operating system I know about. Uh we'll talk about the Let's Encrypt outage and then how Digicert responded to its recent breach. Steve said A plus to Digicert. That's coming up next on se curit now podcasts you love from people you trust twit this is Seurcity Now with Steve Gibson, episode 1078, recorded Tuesday, May 12th, 2026. Digisert does it right . It's time for Security Now, the show where we cover your security, your privacy, and how computers work, and maybe a little sci-fi and vitamin D thrown in with this guy right here. Because basically, this is the show where Steve talks about the stuff he cares most about. Hello, Steve Gibson. I'm not sure if it was self-selecting, but our listeners tend to agree. So they're like there, I mean I'm getting vitamin D email and and and sleep supplement questions, and so I know that our listeners are . And Steve, but we should add coffee to this because Steve is, as we well know, a five-shot venti latte drinker. There it is in the giant mug. I had some wonderful coffee in Kona. They grow Kona uh coffee on the side of the volcano. That's how they named it actually. Yeah. Yes. The name came from the city. But uh it is amazing coffee, so much so that I I bought a large amount of it uh to bring back because it's just it's so no i mean real good coffee is just in a class by itself. But y I don't think you would like Kona coffee, because I remember we talked about this. In fact, this came to mind as I'm drinking it. Because of uh where it's grown uh on the in the volcanic uh soil, it has less caffeine and it and very little bitterness, very little bite. And I remember you like the bite. Yes, in fact, decaf lacks the bite. And so I was like, eh, seems a little why bother me. Kona coffee is is almost like tea. It's very smooth and delicious and a little bit less caffeinated. So yeah, I thought maybe he wouldn't, maybe he wouldn't like it that much, come to think of it. Uh so I didn't send you any. Okay. That's great. I would rather have salt. Expensive. I'd send you salt in my salt side. I do have some from Salt Hank that's coming coming your way. As soon as I figure out how I could package those glass jars uh in a way that they don't break. We had some special salt made for Steve and his wife. They have a a fetish for my son's fault. Yeah, that's right. Okay, so we are giving it away. We are at episode 10 78 for uh May 12th. And I'd I teased this last week. The news was just breaking and I didn't have the whole story. And actually, they didn't have the whole story. I'm talking about a an interesting problem that Digicert , the indust ry's now by far number one certificate authority, suffered. Um I titled today's podcast, Digicert Does it right because I and a lot of other industry experts have singled this their reporting out as this is the way you do this. If you suffer a breach, how do you disclose? And anyway, so we're gonna I I wanna really, you know, give them some props for like the the job that they've done and take a look at what they did s to just uh you know, both give them credit but also to show the in in some detail this is the way it's done right. So many people do it wrong. We should oh we should definitely mention when they do it right. Yeah, exactly. And especially we've covered other CAs that have lost their CA-ness because the of you know trying to like, oh no, that really was oh well you found that. Okay, well, then we'll have to talk. Props to them. Um we're gonna talk about the FCC, however, uh deciding that firmware updates might actually be a good thing. So let's rethink that policy. Uh Netgear, uh speaking of the FCC is the first and, as far as I know, so far only router manufacturer to get a full pass on this ridiculous Euro just got one yesterday. Oh, good. So there's two now. Good. Uh also AI has uncovered a twenty one year old critical remote code execution vulnerability, and this one is trivially implemented in one of the most secure Unixes ever, my favorite and the one I use, FreeBSD. Uh and I should also mention, I didn't get it in the notes for this week, but I'll be talking about it next week. Google has announced that they have uncovered the first AI gener ated zero day. So we now have a confirmed example of AI as we have been worried about, uh, even pre-Mythos. This is not, you know, because presumably the bad guys didn't have access to it, although we do know that there were some leaks of of mythos access, but this has been the concern. So it's beginning to happen. Um there was also a brief let's encrypt outage. We have a another example of a company doing the right thing. Uh turns out, there are now some reports, not surprisingly also, of AI model repositories overflowing with, I'm not sure you call it malware , malprompting. I don't know what mal it's bad, so mal, but anyway, that's the that's a thing now. Uh in addition to you know, NPM and PyPy and everything, all the other open repositories that we've talked about. Although it's a decade old. It's that uh agreement that was signed in twenty fifteen, which allows um uh private companies to share cybersecurity things with the government without fear of reprisal. Looks like that was um uh well, it was temporarily extended. Looks like it's gonna be made permanent. Uh we have some very distressing news about the Edge browser and what was found by someone and has now been confirmed, not only by people online, but some of the my own uh newsgroup participants who have done this and found all their usernames and passwords in the c lear. So we'll talk about that. And then we're going to get into a deep look at how you do this right. If you are a company with serious responsibility for user security , taking responsibility and documenting it. Uh, and Digicert did that. And I'm saying that even though I've wandered away from them, as we know, because of their pricing. But still, they're, you know, they're it. So I think a great podcast for our listeners and of course a fun picture of the week. Which I have not looked at, but will at some point with you after this word. Probably then yes from from our sponsor we will get to that picture of the week in just a moment but first and and to those who are seeing video yes leo has an apparently white shirt on. Oh no, but I have to do it. First time in ever. Oh there was a weird blue stripe hidden behind the microphone. It's a Tommy Bahama Hawaiian shirt. Foliage. Some foliage down below. I guess it do es look like a white shirt, doesn't it? I gotta I might have to retire this from the collection. We're just so used to the collection of orange slices and and I'm gonna go change my shirt after this break. I'll wear something crazy and kooky, I promise. I apologize for not. Our show this hour. Brought to you by Cyber Hoot . If you have ever rolled out security awareness training and thought, this feels more like a compliance exercise and actually teaching security. Well, you're not alone. It kind of is in many cases, many times, you know, platforms try to catch users making mistakes. They send fake phishing emails to inboxes. Then they kind of wait for someone to click and then they pounce on them and assign training after the fact. And let's face it, that can feel uh, you know, punitive. And people don't learn when they're being punished. Honestly, it doesn't change behaviors to do it that way. That's why Cyberhoot does it different. They take a different approach. Instead of trying to trick your users into clicking, Cyberhoot's Hootfish focuses on teaching them first , not in their inbox after a mistake and click, but in their browser, through a trusted realistic phishing simulation. The goal is simple to build instinct before that click ever happens. You're not trying to trap your users. You're you're you're trying to teach them. And by the way, it really works. I w I've watched Lisa go, we've been you we've been using Cyberhood, and I've been watching Lisa go through the training. It really is great. Cyberhood is completely automated. Training campaigns, reminders, escalation to managers, and reporting, that's all handled for you. So instead of you know chasing users, you get clear visibility into who's completed what and where your risks are. And here's something interesting: Cyberhoot also adds a light opt-in social layer. This is kind of cool. Users can connect with coworkers and engage in friendly competition around the training process and how they're doing, right? So this isn't forced gamification. It's just enough to increase participation without turning it into a gotcha system. And users love it. It's fun. A little competition adds a little spice, right? G2 reviewers also love it. They rate cyberhoot I've never seen a rating this good 4.9 out of 5 stars. What the G2 crowd repeatedly praises ease of use, high participation, brief content , quick, non-punitive training, full automation, and strong support. If your organization is ready to stop punishing people for being human and start actually building cyber smart employees, head to cyberho ot.com slash security now. And by the way, please use the code security now at checkout. You'll get twenty percent off your first year. So again, that's C Y B-E-R-H-O- t dot com slash security now and that promo code security now gives you 20% off your first year. Just remember to always laugh, learn, and hoot up. By the way, the sum of the awards are the cutest little owls. I just love it. Cyberhoot .com slash security now. Don't forget the offer code security now . Now, back to our man of the hour, Mr. Speaker. So I gave our picture of the week a title. Okay. I titled it When a Powerful Meme Creates Fertile Ground. A powerful meme creates fertile ground. Fertile ground with apologies to Randall Munro. Oh yes, of course. K C D. All right. Well let me uh let me pull it we'll look at it together for the first time. So you remember his famous X-K C D cartoon with the blocks all teetering on one little block that's written by a single developer. Oh my god. This has gotten a little more complicated. Wow. So this is an updated version of this ridiculous set of towering blocks. Uh hysterical. It's wonderful. I'm not sure if those are supposed to be tombst ones there at the very bottom that with Linus Torvalds, IBM, T S M C and K R. You know, clearly KR is Cernigan and Richie. Um, but at the very, very top, we see a little tiny little black speck that says you are here. And then we have a a zoom in window with a little guy there with that that's whose thought bubble is WTF . And so there's also some guy who's like sort of teetering on the edge that is is labeled web dev sabotaging himself. Uh we've got that all sitting on uh WebSM and there's a V8 engine and it's all bracketed saying something happening in the web , then a bracket on the other side that is more encompassing, uh titled All Modern Digital Infrastructure, referring to all that. We've got Rust devs flying in from the left. It says doing their doing their thing and doing a loop-de-loop and then slamming into the Oracle block. And it looks like maybe J WT, Java web tokens are above that. No, JVM. That's the Java virtual machine. Ah, okay, JVM. And it's teetering. The reason I know that is teetering on Oracle, which owns it, right? Oh boy. Perfect. Perfect. And I love the ang ry bird coming in from right. The right and it's titled Whatever Microsoft is doing is the angry bird. Apparently gonna slam into this whole thing. Uh CrowdStrike's got it its little block. I don't know what I don't know what that means either. Yeah. Anyway, so also wedged in now, we have a new thing. We have what would be a carjack if it were if if it were going going to be a parallelogram and and sort of raise the whole thing. Instead, this is a a a screw based wedge which is expanding uh and that's of course labeled AI because it's threatening to teeter the whole thing off its axis and cause it to all come crashing down. We have that one C99 project bas ed on behavior of undefined behavior. And not to be left out, we've got a cloud flare block and a set of four of the lava lamps, which of course Cloudflare made famous for their for their true random number generating uh system based on the wax in the in the lava lamps. I I also like those those bump bunch of little blue squares in the lower right . I had to figure figure out what I was looking at and I realized it's a shark biting an undersea data cable. Uh it's which of course is gonna cut off the whole a whole chunk of the internet if uh if the if it chewed through the the the cable ling on the ocean floor live curl not to be forgotten is there aws uh uh c developers writing dynamic arrays and we are reminded that all of this is driven by made possible by electricity so they the the underlying foundation is a big block of electricity with some you know electric poles coming in to feed it. Very, very funny. This is this is not only funny, but really pretty true. There's a lot of yeah. This is all the stuff we've talked about through the years on the podcast. Yeah. Wow. Very nice. So a fun picture. And thank you to our listener who sent it to me. Okay, so there's news on the residential router front. Apparently , someone at the FCC got a clue, or at least they listened to somebody who actually knew something about cybersecurity, because last Friday they announced a reversal to their previous no updates for you policy. Tom's hardware covered this and wrote the Federal Communications Commission announced on Friday, May 8th, through its Office of Engineering and Technology, the O ET, that it was extending temporary waivers, allowing certain foreign-produced drones, drone components, and consumer routers to continue receiv you know all those bad things that we think are coming from China to continue receiving software and firmware updates in the United States. In late 2025 they remind us and early 2026. The FCC added these categories of equipment to its so-called covered list, which effectively blocked already authorized devices from receiving post-approval, software and, firmware modifications. The agency subsequently issued waivers permitting critical security and functionality updates to continue through March 1st of 2027. So here we are around the same time in 2026. So basically, you get a year more of updates for consumer routers. Now, under the now updated waiver, manufacturers of affected devices will be allowed to continue issuing software and firmware updates until at least the first of January 2029 . So almost another two years, provided that the devices had already been authorized for use in the US before being added to the FCC's covered list. Meaning nothing new can come in. We talked about that before. No new model numbers, which again is n uts, but okay. They write the extension also broadens the waiver to include certain class two permissive changes involving software and firmware updates intended to mitigate consumer harm. In its notice, they wrote, the FCC acknowledged that continued software support remains necessary. This is them, you know, I the light turning on for them. Hey, uh continued software support remains necessary to protect US consumers. What do you know? The waiver specifically allows updates that maintain device functionality, patch vulnerabilities, and preserve compatibility with changing operating systems and network environments. But still no new models. But oh, you can have all the firmware updates you want. Which again, what? The agency argued that the public interest would be better served by allowing these limited updates rather than freezing software support entirely. Okay, in other words, duh anyway, t Tom's adds, the waiver does not reverse the broader restrictions or remove the devices from the covered list. It only it applies only to already authorized products and the software and firmware related changes intended to maintain safe and secure operation, manufacturers must still comply with other FCC requirements governing permissive changes and equipment certification. Okay. So as I said, given January 1st, 2029, that allows for nearly an additional two years of updates to existing routers, which yeah, that's certainly good news. But of course, the entire thing remains unspeakably ridiculous because control over a router's firmware is all anyone needs to turn that previously authorized and approved because it existed back a year ago , router, uh, into an internet bandwidth weapon. The hardware doesn't need to change, model doesn't model number doesn't need to change, it's all firmware. So either you trust the foreign manufacturer of a router or you don't. And if you do, then there's no problem. And if you don't, then limiting updates , like allowing any updates, but you know, limiting them to the original March 1st, 2027 deadline, even that, one year, is of absolutely zero benefit since you've given them under the assumption that they have malicious intent one full year to cook up some new sneaky malware update with which to infect any routers that may be updated during the period of that year. In other words, none of this has ever made any kind of sense . As we saw uh last week, CISA, our CISA agency has been effectively neutered. Our agencies appear to now be staffed and run by people who will not push back against policies that they know are clearly wrong. So this sort of nonsense is what results. It's difficult to imagine this could have happened back when you know CISIR was at its original strength and staffing. But you know, because there would have been people there would have said, what? No. I mean, there , they one of the reasons we liked SISA so much was that they had taken such they had taken such responsibility for getting into the you must update your stuff business and pushing that out to all of the government agencies over which they had any oversight . Apparently that's not what we do anymore, even though it was the right thing to be doing then. One piece of good news, and Leo, you added a second piece of good news, uh, for thy for those who like and use Netgear routers and now the Eero products, is that even before the addition of those additional two years of firmware updates, Netgear, and now we know Eero had announced that it had received, they had received now the FCC's conditional approval for their routers. This meant that none of those ridiculous FCC imposed restrictions would affect any of Netgear's and Eros router products, not those already sold and not any current or future models. It's like this, you know this membership on this uh list just doesn't exist. So they get a full pass, uh, which also includes their right to update their firmware with abandon anytime they feel the need. So yay . Okay, so uh we've heard again from the guys at IELTS Security. Remember their name, A-I S L E . They're that commercial group who've been using their own AI, as their name suggests, to find flaws in software and who were somewhat annoyed, as we discussed a couple weeks ago, by all the hoopla that Anthropic was able to generate around Mythos . The headline of last Thursday's posting of theirs was IELTS Discover's CVE 2026 42511, a 21-year-old free BSD remote code execution vulnerability . So AI was used. This thing has been in this been a problem in free BSD for 21 years. It actually, as we'll see in a second, actually inherited it from OpenBSD when it when it went when the open source pro one one open source project, FreeBSD, grabbed another chunk of code from a different open source project, open BSD, and with it came a serious problem. So this was this posting of I L was written by the discoverer of this flaw. He writes free BSD is often described as one of the most secure operating systems in the world, with its reputation arising from its high-quality networking stack, deliberate engineering, and a philosophy of security through simplicity, FreeBSD's history and usage are remarkable. It powers Netflix, open connect infrastructure, Sony's PlayStation OS, part of Nintendo 's Switch OS, Yahoo's backend services, NetApp's storage systems, Citrix's Netscaler, has long helped form the software base of major networking platforms , Cisco, Juniper, and so on. WhatsApp's back-end services historically, and is now the focus of a substantial foundation effort to make it work better on modern laptops. And he writes for full disclosure, remains this author's personal operating system of choice . And to that I will just add that it's also my own Unix OS of choice, as I've often mentioned. It underlies the PF Sense personal firewall router system . Uh, and for me, it runs our DNS and our news groups. So, you know, that's the Uni x that I chose. Uh uh , you may remember, Leo, years ago, a guy named Brett Glass was uh active in the early days of the PC uh industry. Uh and Brett knew his way around Unixes. And I remember having a conversation with him and saying, So what do you recommend? He said, free BSD, period. There are other BSDs. Th'eres net BSD as an open BSD, but free BSD is the one you like. I do. Uh and it has had some some desktop uh laptop um orientation and some a lot I think it's seven hundred fifty thousand dollars from the uh some foundation affiliated with free bsd are making a serious push to make it more uh desktop and laptop friendly, adding a lot more Wi-Fi drivers and and and make making it a lot more hardware agnostic. So it's it's still alive and kicking. Anyway, I'll continue saying I 'll discovered a remote a remote command execution vulnerability in FreeBSD's DH cli ent that is trivially weaponizable and wormable, yikes, by any system on the same local network as the FreeBSD system. The vulnerability first entered FreeBSD in the 2005 , that's the year we started this podcast. 2005 release that's 21 years old, and so is a twenty so is this podcast. 2005 release of free BSD 6.0 when open BSD 's DH client was imported and laid dormant, that is the vulnerability did, until discovered by IELT. The vulnerability also affected OpenBSD until 2012, when that operating system deprecated DH client hyphen script completely, indirectly fixing the vulnerability. But FreeBSD didn't. The initial flaw was identified by IELTS AI-based source code analysis pipeline and then investigated by our triage agents. Joshua Rogers, that's actually the author, so he's referring to himself, of IELTS offensive security research team traced the relevant code paths, established the full security impact, and developed a proof of concept demonstrating a complete local network to root exploit chain. FreeBSD is adding key improvements to laptop support, including greater Wi-Fi support, so the attack surface here becomes even more relevant to everyday systems. A malicious wireless access point, or in some cases another attacker just sharing the same Wi-Fi network able to spoof DHCP can target the exact D HCP path that almost every wireless free BSD system will rely upon. Imagine you're the author of this post, who runs free BS D on their laptop, as this guy does, you're at a coffee shop, airport, or hotel. And as soon as you connect your free BSD equipped laptop to the Wi-Fi, your whole system is hij acked in secret. Imagine you have a PlayStation whose OS is locked down from any unofficial access, only to be hijacked by connecting to a network. In other words, this vulnerability not only affects servers, but any free BSD machine that connects to a network using DHCP, which is the default setup case for almost everybody . The vulnerability was a logic flaw that allowed attacker control led protocol data to be persisted into a trusted configuration like format without proper sanitization, then later reinterpreted in a privileged execution path. Autonomous security platform is built to find. And get how he signs off here. He says, like our recent findings in OpenSSL, Firefox, Live PNG, and Amazon's crypto stack. This result came from disciplined engineering and end-to-end analysis , not model mythology. Oh, please. So okay. Sounds like they may still be somewhat annoyed by the mythology of mythos. Which is not mythology as you pointed out. Exactly . It's happening. But in any event, AI truly is finding serious flaws, many of which have been present for decades. In this case, we're talking about FreeBSD's DHCP client , which can be fed a maliciously formed reply DHCP reply containing code or commands and code that it will execute. As Joshua, who authored this write up, noted, this could have been extremely serious if it had not been found by the good guys . At some point, we may see those who claim that AI enhanced software vulnerability discovery never turned out to be such a big deal. Uh remember though, that's what Y2K could have been a big problem if it hadn't been caught beforehand and dealt with. Objective observers would, I think, do well to remember all of the many critical vulnerability discovers discoveries like this one that did serve to clean up our archaeological code base before the bad guys had the chance to get in there and exploit it. The question of course how long it's going to be before the bad guys get access to these models. Well, you're gonna have a story about this in just a little bit, actually. Yeah. Yeah. Okay. So there was a bunch of internet chatter last Friday about a several hours outage of Let's Encrypt. Our listeners know that with so much of the web now utterly dependent upon certificates issued by Let's Encrypt, and with maximum certificate lifetimes continuing to drop, and especially with Let's Encrypt's optional super short six-day certificates now being available. Any outage of the system upon which so much now depends is of interest. And I get and I'll say again that you know unfortunately it all of this I mean I get how let's encrypt happened. I understand the appeal , but it it's so antithetical to the the deliberately distributed model of the internet . I I hope this never comes back to bite us because it is creating a single point of failure where everything else has been designed to prevent that. In this case, uh, this outage, such as it was, was deliberate and temporary it was an administrative suspension of new certificate issuance following reports of a missing extension from one class of certificates. Let's encrypt's post-incident report said, and you know, this involves a lot of inside baseball terminology. They said Let's Encrypt's Gen Y , which is the YE and Y R Cross Certified Subordinate CAs , were issued in violation of CCADB policy, which requires that the server off EKU extension must all caps must be present in cross signed intermediate certificates issued since J Okay, so the certificate extension in question, uh which is to say a server off EKU, where EKU is the abbreviation for extended key usage , it specifies the it it specifies limits, the imposition of limits on the application of any certificates which that CA intermediate certificate would be validating. And so these limits are things like can be used for server authentication or client authentication, you know, and/or code signing and or email protection and or time stamping so again it it it l it it it specif ies what that certificate is authenticating for what purposes ? Um , and that extension was missing. So it should have been there. It wasn't. It mostly doesn't matter that it wasn't, but as we know, any certificate authority must again, I should be all caps, M U S T, take their responsibilities absolutely seriously. And let's encrypt did. They immediately stopped issuing new certificates when this then when this issue came to my came to their attention and they confirmed it. They fixed the problem. They resumed reissuing or resumed issuing certificates . In their words quote, we temporarily disabled certificate issuance, deployed a configuration change to prevent future issuance from the cross-signed gen Y hierarchy and then re-enabled issuance. So thank you very much. We fixed it. Problem solved. Nothing to see. But again, they are, you know taking their responsibilities, the the authorization, essentially the trust that the entire industry is placing now on 70% and growing of all web certificates which are being signed by Let's Encrypt, we need to know that you know they're doing that job correctly . And Leo, we're a little after uh a little past half half an hour in. Let's take a break and then look at uh unfortunately why we can't have nice things. The yes, the poisoning of AI models. Pulling models from open claw and uh the uh I use open router, uh Hugging Face is and I know Hugging Face. You have you have to know that because Hugging Face has more than a million models. Yes. Where'd they come from ? Yeah. People are just making them. Do you need them? Oh no. Well, you need some . I don't know if I need the bad ones. We'll find out what that is in just a little bit, but first a word. From Guard Square , our sponsor for this segment of security now. Now, are you a mobile app developer? You really want to listen up here. Mobile apps today are, and this is good news for you as a mobile app developer, an inescapable part of life. We use mobile apps for everything from financial services to healthcare, to retail, entertainment. And here's the thing: us ers trust those mobile apps with their most sensitive personal data, especially in finance and healthcare, right? Unfortunately, a recent survey showed 72% of organizations have experienced a mobile application security incident in the past year. Ninety-two percent of respondents reported rising threat levels over the last two years. I think you you don't need a survey to tell you that . Meanwhile, attackers who They're constantly finding new ways to attack your mobile app. I'll give you one way that they do it. For instance, they will take your app . It's actually fairly trivial now with AI and Ghidra to reverse engineer it, take it apart, repackage it. By the way, this is I suspect what's happening with these AI models as well, repackaging it, and then distribute the modified app. They do it with uh phishing campaigns. Hey, we got an update. You'd got to get the latest version. Or side loading or third-party app stores. There's all sorts of ways to do this. That's just one of many ways they're attacking your app. But you need to prevent this. You need to take a proactive approach to mobile app security. And by doing so, you can stay one step ahead of these attacks. And more important,ly most importantly, maybe maintain the trust of your users. And that's where GuardSquare, our sponsor, comes in. Guardsquare delivers mobile app security without compromise, providing advanced protections, both Android and iOS apps, combined with automated mobile application security testing to find vulnerabilities, and real-time threat monitoring so that you can gain insight into the attacks that are occurring, not you know, in general and to your app specifically. Discover more about how GuardSquare provides industry leading security for your mobile apps at guardsquare.com. That's guardsquare .com . We thank them so much for supporting our show and for the work that they're doing to uh protect us as mobile app users and mobile app developers. Guardsquare.com. Steve, I've replaced the white shirt as you can see with a shirt. Ah. Now we recognize you. I got this in Orlando when we were out there for the Gator Trust World. Yeah, this is my uh from Gatorland. It's got gators on it. Yep. Nice. Little bit more recognizable. Not not a white shirt for sure. All right. Now I want to hear about this AI thing. Oh boy. So last Friday, the next web posted the news of an analysis of the large language models being hosted and offered at Hugging Face and Claw Hub . Uh, the news is not good. Yeah. Here's what they said. They said the two most important software supply chains in artificial intelligence have been systematically compromised. Hugging face , the repository that hosts more than a million machine more than a million. Where the what? It's amazing when you go there. It's amazing . I mean, there are, you know, the the kind of maybe several dozen root AI models, but then people are creating their own spins of it and so forth. It's really uh incredible. Repositor y that hosts more than a million machine learning models used by virtually every AI company on the planet has been found to contain hundreds of malicious models capable of executing arbitrary code on the machines of anyone who downloads them. ClawHub, the public registry for open clause AI agent skills, has been infiltrated by a coordinated campaign that planted three hundred and forty one malicious skills designed to steal credentials, open reverse shells, and hijacked The attacks are different in technique but identical in logic. Both exploit the implicit trust that developers place in shared repositories. Both use the infrastructure that the AI industry built to excuse me AI industry built to accelerate development as the vector for compromising it. Hugging face has been aware of malicious models on its platform since at least 2024, when security firms JFrog and Reversing Labs independently identified models con taining hidden backdoors . My turn to have a throat tickle, sorry. Yeah, I'm just looking right now at uh hugging Face at their model repository and think this is actually kind of stunning. They list two million eight hundred sixty nine thousand and eighty six different models. My God. Yeah. I mean's a well, let hope they have a good search engine because they do, actually. They have a very good search engine. Wow. And and the thing is, it's not every mod, you know, there it's it's not like uh chat GPT alone. I mean, there are models to do all sorts of uh things. I I have a specific model I use from Hugging Face that's just for text embedding. That's all it does. So, you know, and these are highly customized in many cases. So so so very vertical application. Very vertical. Exactly. Exactly . Yep. I mean it's a it's a great repository. This is really somewhat different though from the OpenClaw registry. But I'll I'll let you talk about this because I know. So they said Hugging Face has been aware of malicious models on its platform since at least at least 2024, when security firms JFrog and Reversing Labs independently identified models containing hidden backdoors. The problem has not been contained, it has scaled . Protect AI, which partnered with Hugging Face to scan the platform's model library, and given its size, that's no small feat, has examined more than four million models and identified approximately three hundred and fifty-two thousand unsa fe or suspicious models across 51, 700 models. I'm sorry, three hundred and fifty-two thousand unsafe or suspicious issues acrossed fifty-one thousand seven hundred models . J Frog found more than one hundred models capable of arbitrary code execution. The attack technique known as null if AI , nullify, null if kind of you know play on nullify, null if AI, exploits Python's pickle serialization format, the standard method for packaging machine learning models . Attackers embed malicious Python code at the start of the pickle byte stream and compress the file using 7Z rather than the default zip format , which breaks Hugging Face's pickle scan detection tool. Well, and that's just dumb that Hugging Face can't check seven Z compression in addition to zip. The payloads are not subtle, they write. Security researchers have documented models that establish reverse shells. Okay, meaning that it connects out to a remote command and control server and says, What do you like me to do now ? Connecting to hard-coded IP addresses, giving attackers direct access to the machine of anyone who loads the model. Others execute credential theft, exfiltrate environment variables, or download secondary malware to the user's machine. A data scientist who downloads what appears to be a legitimate model for a research project or production pipeline is in some cases handing control of their machine to an attacker . Hugging Face has responded by partnering with J Frog and Wiz Security to improve scanning capabilities. Remember that Google bought Wiz. JFrog's integration has elimin ated 96 % of false positives in Vulnerability. Anyone can upload a model. The scanning catches known patterns. The attackers who designed Nullif AI built their technique specifically to evade that scanning. ClawHub, the registry for OpenClaw's AI agent ecosystem, faces a different but related problem. OpenClaw has grown to three point two million users and a And attracted partnerships with open AI, but it but its skill registry has become a target for attackers who understand that an AI agent executing a malicious skill has access to whatever the agent has access to, which in enterprise environments can mean databases, APIs, internal networks, and cloud credentials. In other words, we're giving agents in order for the agent to have agency, it we need to give it control and access to things. Unfortunately, the malicious skill inherits that co i security audited all 2, 8 5 7 skills, thank goodness that's a manageable number, on Claw Hub and found, unfortunately, 3 4 forty one malicious entries. Of those, three hundred and thirty five were traced to a single coordinated operation called claw havoc. Separ ately, Sin x, you know, S N Y K, Sinc's Toxic Skills Research examined the broader ecosystem and found that 36% of all say better than one out of three, thirty-six percent of all AI agent skills contain security flaws with approximately nine hundred skills, roughly twenty percent of the total, classified as malicious. So one in five, deliberately malicious. Thirty skills from a single author were silently co-opting AI agents for cryptocurrency mining. You know, which right makes sense. You got a super powerful GPU. You got, wow, that AI is really working hard for me. No, it's working hard mining cryptocurrency for somebody else. They write the claw hub the claw hub attacks are particularly dangerous because of the nature of AI agent architectures. The rise of model context protocol and similar standards in the agentic era has created a new category of software supply chain in which AI systems autonomously select and execute tools from external registries . A compromised skill does not require a human to click a link or open a file. It requires an AI agent to select the skill The hugging face and claw hub compromises are the AI specific manifestations of a supply chain attack pattern that's been accelerating across the entire software industry. In March of 2026, the Light LLM package on PyPy or PyPI was compromised, potentially exposing half a million credentials, including API keys for Meta, OpenAI, and Anthropic. Meta froze its AI data work after the breach put training secrets at risk. In April, a Bitwarden, as we know, we covered it, command line uh instruction package on NPM was hijacked for 90 minutes with a payload specifically designed to harvest credentials from AI coding tools, including Claude Code, Cursor, Codec CLI, and ADER. Days later, the Pi Torch Lightning Pro package was compromised for forty two minutes with a credential stealing payload from the mini Shy Halud campaign. The European Commission itself was breached after attackers poisoned Trivi , and we've talked about that, an open source security scanning tool, demonstrating that even the tools designed to detect supply chain attacks can become vectors themselves for them. The United States Department of Defense published formal guidance on AI and machine learning supply chain risks in March of 2026, acknowledging at an institutional level that the AI software ecosystem has become a national security concern. The common thread is spe ed. The PyTorch Lightning compromise lasted forty-two minutes. The Bitwarden CLI hij ack lasted ninety minutes. The light LLM attack window is estimated at hours. These are not persistent campaigns that defenders have weeks to detect. They're brief, targeted insertions that exploit the automated dependency resolution systems that modern software development relies on. A developer who runs a package install at the wrong moment downloads the compromised version. The window closes, but the damage is done. The AI industry has invested hundreds of billions of dollars in model training, inference infrastructure, and application development. The investment in secur ing the repositories through which that software is distributed has been a fraction of the total. Hugging Face has partnered with security firms. ClawHub has implemented basic moderation , package registries, have added two factor authentication requirements. None of these measures has presented the attacks documented above . State actors can already produce AI powered malware that evades conventional detection, and the supply chain attacks on AI repositories represent a natural evolution of that capability. The models and skills hosted on Hugging Face and ClawHub are consumed by systems that make autom ated decisions, process sensitive data, and operate with elevated permissions. A compromised model in a production AI pipeline is not equivalent to a virus on a personal computer. It is a backdoor into an automated decision-making system that the organization trusts precisely because it appears to be a legitimate component of its AI stack. The fundamental problem here is architectural. The AI industry bu ilt its development infrastructure on the same open registry model that has defined software development for the past two decades. Centralized repositories where anyone can publish automated tools that download and execute code from those repositories, and a culture of trust that treats popular packages and models as implicitly safe. The difference is that AI models are not just code. They're serialized objects that execute during deserialization, a property that makes pickl pesickle based models inherently more dangerous than traditional software packages because the malicious code runs the moment the model is loaded before any human has a chance to inspect it. The AI supply chain is now the most attractive target in the software security space. The repositories are trusted, the consumers are automated, the payloads execute on load, and the industry that built these systems is spending its security budget on model alignment and prompt injection, while the infrastructure through which the models are distributed remains in the assessment of every major security firm that has examined it comprehensively compromised So Leo, I would say that the caution and trepid ation you felt and shared when you were first considering turning open claw loose in your world was likely warranted. Um this is not an entirely new yeah this is not an entirely new phenomenon although with popularity comes you know increased focus by the bad guys and we've seen this thing just skyrocket over you know so far this this year. The difference between the two Hugging Face and the skills repository is it's trivially easy to write skills. They're just text. You know, to make your own malware model, that takes a little bit more skill. But anybody can write a skill. It's just plain text. And so though I mean, I guess the skill involved is is how you insert the, you know, Shy Halud style, you know, Bitcoin stuff. But um it's it's not complicated. And so that's why I think you're gonna see in these in these registries for skills. I never use a skill uh from uh the public. I look at skills. What I often do, and I would recommend people do this, is I point my as a as a training base. Yeah, I point my assistant. I say, here's a GitHub repository for a sk ill. Assess this. Tell me what you think of it and how we could apply it. And then let it write a skill, which I will then check. Uh, and that actually works quite well. I've built basically my own open claw from scratch with just the pieces that I want. And it's a lot it's I think. I love that you said how we could apply it because you didn't refer you weren't talking about you and Lisa. You were talking about you and the AI. Me and Claudia, my good friend. It is so difficult not to think of this thing as an entity. I mean, it is just astonishing. Anyway, I wanted to share this with our listeners because I'm certain that that you know we have many listeners who are enjoying playing around with and experimenting with and perhaps even deploying systems or solutions using these openly available AI models . Please, please, please be careful. Because one of the problems here is this makes it the way this has been built and deployed makes it so easy to use this stuff. That just I mean, that that alone should be a cause for raising a red flag uh in you know in in the minds of any security aware person. So you know, as as you are, Leo, you know, you are, you know, looking at this and not just saying, go, you're saying, you know, let's take a look at it. What should we do? And well, you've taught me, sensei, over the many years that we have done this. I'm glad it's sunk in. That's good. But again, it's so easy in a moment of enthusiasm to say, well, and you were tempted, right? When when open claw happened, it's like, oh, should I, shouldn't I? But you know, somebody else is not gonna be, you know, who hasn't been sitting here for the last 21 years with me, is going to go, hey, this is great. Right. Right. Yeah. Uh if you have any nervousness about it, trust your instinct because you're right. Yes. Basically. Yes. So there's some good news on the horizon. The word is that the original decade-long CISA, that's CISA 2015, stands for Cybersecurity Information Sharing Act, which, as we have talked about on a number of occasions, expired last year because it had a decade-long uh life from 201 5 to 202 5 , uh, which was then temporarily extended until this coming September, is now in the process of receiving its much needed long-term reauthorization. And remember that this is what allows private sector enterprises to share s their cyber intelligence with the government without fear of any legal blowback or reprisals. So this gives them cover. And we've had we've heard from CEOs and CIOs who've been saying, you know, you know, we we we really need to share some stuff that we have like it's important , but we can't because you know we we we can't risk having ourselves you know taken to court. So anyway that that hopefully another 10, another decade worth of coverage is coming soon . Uh, a number of our listeners pointed me at the news that Microsoft's edge browser uh uh is doing very little, much much less than it could, to protect its users' passwords. A posting from the Sands Institute, which I've enhanced a bit, reads Yep , it's for real . Uh the posting wrote this started with a post in X which highlighted research by and we have a an at sign handle that's clearly hackerized living off the LAN , uh, with you know ones and O's and zeros and threes. I don't know I see a 12-year-old. Uh uh, but that person did find this issue the uh the the um the sans te posting said ed ge stores all of your browser passwords in clear text . Oh great . Okay. Yep. Oh great. Even if you have not used them in this session, you know, just in case., he writes You might want em. He said, I figured it couldn't be that easy, right? But like so many things, yes, yes it was. To reproduce this open edge. Don't browse anywhere. Just open it. He says flip out to task manager. Search for Edge. Then expand that task. Highlight the browser subtask , right-click and choose create memory dump. Navigate to where the DMP file is stored. If you have not used strings before, you're in for a treat. Strings is of course just part of most Linux distros, but you can easily get a copy for Windows as part of MS Sys Intern als. Now let's look for passwords. You could use strings and look for known credentials, just search for a known password and you will certainly find it. Or you can take advantage of the format of the saved data, which is the URL of the site, followed by its protocol, meaning like HTTPS probably, then a space, then you the user ID, a space, and the password. All of that for the site. He says, so searching for TLD protocol, right? Mean for example, Google.com immediately followed by HTTPS . He says, which in most cases is like, oh, or just comm, just C O M H T D P S with no spaces. He says we'll find them and they'll all be in one nicely formatted group, no less. The command for that will be strings space hyphen N space eight space ms edge dot DMP and then a space and then a vertical bar for piping, then find double quote C O M H T D P S double quote and then hit enter. Bang. He says it really is that easy. And the ironic thing? To view these same credentials in the browser, there's a whole security theater process where Edge wants your biometrics as proof before disclosing even the user ID and site names. You know, for security. All the while, the whole shot is there in clear text free for the looking. Also is noted in the X post, Microsoft classifies this as intended behavior. I'm not sure what manager or lawyer, he writes, decided that. Hopefully it wasn't anyone in their security team. Any logged in Windows Edge user can dump all of their stored edge credenti als with no additional rights, which means any malware that the user executes also has access to all of those credenti als for the asking. But he says not to worry, right? It's intended behavior. Remember this is what Chrome did for a while. Well it's it's Chromium. Edges Chromium based. We I don't think they do now though. That's what's surprising. But I'm not sure And he said it's intended behavior if what's intended is also to get me to use Firefox or Chrome. Yes. It's working . Gosh. So and I did uh upon uh that coming to the attention of some of the guys in our security now news group uh someone thought really and did it and he's like oh crap yes terrible there's all of my username dom domains usernames, and passwords. Bicasally , you you get that, you can log in as that person anywhere . Wow . Crazy. Okay, we're at an hour. We're gonna do some feedback, Leo. Let's take another break and we will hear from two of our listeners. Absolutely. I can figure out what button I need. Oh, there, okay. I need to press. Uh it is it is time to talk about Doppel. As in uh as you pointed out a couple of weeks ago. Doppelg as in Ganger. Yes, as Steve said . Doppel, our sponsor for this segment on security now. Uh maybe you know that message you just got, maybe that is an urgent message from your CEO, or could it be a deep fake trying to target your business? I think these days, more likely the latter, AI can impersonate trusted individuals. I mean, down to the voice. Down to the they can do it in video now. Doppel's platform illustrates how frequently users fall for these phishing attempts. They call it phishing, right? With a V. In voice call simulation deployments, target users . They didn't know they were target users, right? This was a test. They spent six minutes conversing with a deep fake. And then afterwards, when they talked to them, a hundred percent of them thought, oh no, that's a that that's human. It's not, it was an AI . But they didn't believe it. Doppel is the AI native social engineering defense platform. Doppel strengthens human risk management by training employees to recognize deception while their digital risk protection detects and disrupts attacks across every channel. As attackers turn to AI to power increasingly sophisticated strikes, Doppel uses AI to fight back with automated tak edowns, multi-channel coverage. Yeah, because it's not just email anymore, any, is it? It's text, it's Slack, it's voice. And and and Doppel builds AI defenses that put intelligence into every fight . Intelligence in your users too, that builds intelligence with them. Doppel works relentlessly to protect people, brands, and trust. Doppel. D-O-P-P-E-L. Best in class integrations and partnerships to seamlessly integrate into the existing security stack you're using. So that's nice. And Doppels Industry Awards and Testimonials speak for themselves. They're recognized as a winter 2026 G2 leader in Users Most Likely to Recommend, Momentum Leader, and Best Support. Kind of the big three. Join hundreds of companies already using Doppel to protect their brand and people from social engineering attacks. It's out there, it's happening right now. You need Doppel. Doppel outpacing what's next in social engineering. Learn more at Doppel. com. That's D-O-P-P-E-L dot com. Doppel . Ah It's a shame we we have to do this now. But it the attacks are getting so sophisticated, it's uh it's it's really mind bending. Yeah, the ultimate uh consequence of the cat and mouse back and forth with with escalation on both sides. Yeah, that that's actually right. Yeah. On we go with the show, Mr. Gibbs. So Todd Whitaker, our listener, writes, Steve, I thought this might be of interest for security now. The rival security group has a thoughtful follow-up on the Claude Mythos free BSD exploit story, arguing that Mythos may not have been quite as creative as the initial coverage suggested. And he has a link to their uh their uh whole posting. He writes, their claim, as I understand it, is not that the result is unimportant, it is that the vulnerability, the prior fixed pattern and perhaps even the exploit relevant structure may already have existed in the model's training data. The free BSD issue appears closely related to CVE two thousand seven three thirty nine ninety nine in uh occurring in MIT uh , the same general RPC SECGSS validation logic, same stack buffer overflow pattern, and a strikingly similar bounce check fix. So, Mythos may have discovered something genuinely dangerous, but perhaps by recombining known historical material rather than reasoning from first principles in the way He writes that still seems worrying, just in a different way. If advanced models can rediscover old vulnerability patterns embedded in the fossil record of open source code, then attackers may not need models to be brilliant. They only need them to be tireless, well tooled, and good at recognizing dangerous old ideas in new pla ces . And I'm going to interrupt because Todd has a little more bit more to say about something different. But I just want to say I completely agree with that. As our understanding of computer science has evolved . One of the things that's happened, and it becomes it's kind of gone into the parlance now of computer science. We've come to notice patterns in the solutions to problems. They're like a short and small abstraction away from the concrete solution where we see that many different such concrete solutions can be grouped together by their sharing of a common underlying pattern. So what rival security observed was mythos preview finding what we might describe as a flaw design pattern, a common type of mistake that coders have been making through the years, which winds up being a natural sort of mistake to make due to the underlying architecture of the computer behavior that we're programming. We all know that today's large language models excel at pattern discovery. Probably more than anything, that's what they are. So we would expect that if someone somewhere made a similar mistake and its correction that had been captured in the model's training corpus, then it would indeed be able to make the connection. So I'd say that this is an interesting and useful observation about the underlying way, in this instance, mythos discovered maybe rediscovered, you know, the newer similar patterned flaw. But I don't like Todd, I don't see anything taking away from the fact that, you know, as Yoda might say, discover that flaw it did. Yeah, I mean it's not like it knew about the flaw, it just recognized the pattern. I mean, yeah, it said, uh this seems familiar. It's like if you recognized a buffer overflow. I mean Exactly. Yeah. Exactly. So his note continues with some interesting observations from his own background about AI as a computer science educator. He writes, I would also be interested in your broader take on what this means for computer science education and the profession. My current working view is that the most productive human AI collaboration in software engineering depends on advanced judgment, architecture, design patterns, there's patterns, threat modeling, failure modes, invariance, trade-off analysis, and knowing when the AI's answer is superficially plausible but structurally wrong. The problem is not that students must learn the old ways before they're permitted to use the new tools. He says that's just reframing the old learn assembly before C ument. The real issue is that AI makes coding cheaper while making judgment more valuable. He said to supervise AI, to supervise AI generated software. A person needs mental models. How state behaves, where abstractions leak, how protocols fail, why concurrency is treacherous, how memory and parsing bugs become security bugs, and why a working MVP may still be architecturally unsound . Those mental models do not emerge from prompting alone. If we let AI collapse the difficult apprenticeship too early, we may produce developers who can ask for software but cannot reliably tell whether the software they received is safe, coherent, or professionally defensible. He says, I'm writing from my personal email, but my day job is in computer science education . So this one lands close to home. I spend a fair amount of time thinking about what we should still teach humans when machines can increasingly produce the code. Which so cool feedback, Todd. Thank you. I don't think I've I've seen any more coherent and clear description of the AI versus human coding question. And I love his one line. The real issue is that AI makes coding cheaper while making judgment more valuable. And I think that captures where we're headed in computer science education and vocation, in the same way that any higher level language lifts its coder away from the grubby details of the specific underlying computer hardw are, the use of coding trained large language models clearly lifts its users from the grubby details of the way computers are applied to problem solving . As many, you know, never before programmed anything users are discovering, they're now able to simply ask for what they want the computer to do, and the LLM will almost magically produce a potion that does that. But there are clear limits to what can be asked for. We saw a perfect example of such lim its last week when those bad guys who had created that credit card clearing web portal apparently just forgot to ask for authentication to be added. Whoopsie . What we're seeing rapidly evolve during the rush to use AI for code generation is that for AI to be applied to the creation of any very large and complex solution, a solution architect is still required. No large problem can be dropped whole into AI's lap. At least not today, not yet. And I don't know when . Instead, for now, today, a solution architect who's trained and experienced in the application of the various higher level solution abstractions that have been developed over the years of true computer science needs to carefully decompose These solution architects are the true core of the science of computing. You know, coding is just their implementation. And these are the sorts of things, you know, that Donald Knuth and other scholars of the art have spent their lives exploring and documenting . So yeah, I think Leo, what and this is what you've talked about, like the way you're now approaching the application of AI is because you under stand about the way computers are applied, you're breaking the problem down into pieces that you you intuit AI is able to do the the the the grunt work on and but you're but you're you know giving it the interfaces that it needs for the various pieces of individual grunt work. Yeah and I'm finding more and more uh well I I do the coding uh for as a basis for things like cron jobs, like things that are gonna run over and over again. And then a lot of what I'm using AI for now is text uh-based stuff. I had it plan our itinerary for uh Hawaii, for instance. And uh if if you give it the basis , uh the information it needs, I have it use my uh obsidian journals and things. It does a it does really a quite a good job. I sent it to my travel agent. I don't know how thrilled she was with the generated recommendations. I I wanted her I wanted her to say whether that was a good said they were good or not. Uh, but I realized it w she might feel like it was kind of taking her job a little bit. I think it's probably going to. Yeah. Yeah. I mean, there's something that she does that no AI could do, which is the relationship she has with the various vendors. Yes. Yes. And that's uh very you know, that's human and only And things that she has heard from her other clients. She where it's like, you know, I heard about this this you know, you want to make sure you you spend more time at on, you know, at this port because I did. I reassured her. I said, you know, this is this is a nice starting point, but the it can't duplicate what you do as a human being. And I think that's really the lesson that all of us should learn to calm down about AI is that we we st we still need humans. Humans add something that no AI can can can do. Or will I think we'll ever be able to do. Yeah, what I have found, I I I remember very early on I shared one of my prompts where I was, you know, I went on at some length and I remember, you know, you were surprised that I was talking to it as much. But if the more language you give it, because it is a language engine, the more language, you know, descriptive language you give it, the more it has to work with. Yes. Yeah. Uh I've found I'm writing more more and detailed specs uh and plans than uh than ever because it does help it be more accurate if you're very clear about what you want. Yeah. Okay. Next listener, Randy Crumb says Hi Steve. In episode 1077 last week, you mentioned you think companies that have closed source software should move quickly to utilize anthropics mythos, clawed security or, similar to ols as they emerge. Their closed source code is only closed sourced to the outside world. I think you glossed over the risk that using these online AI tools potentially ex poses your closed source code to the world. They have privacy and security tools in place, sure, but their motivation is financial. They have a setting to disallow using your AI conversations to train their AI models, but you have to trust that they're actually following that setting. For the same reason you trust Apple with your data more than Facebook or Google, the use of online AI tools to review closed source code is a risk. A security breach, internal or external, could be devastating to a software company that has their code exposed. He writes, I've been expecting with LM Studio to run local, offline, open source LLM models for use with proprietary data. He says, parents note I work with client data, not code. The hypothesis is a local offline LLM can be safely used with confidential internal proprietary data or code. These LLM models are usually not as current as the online tools, but they catch up quickly. Also, they're not as flashy, newsworthy, or marketing hyped as strongly as the major online tools. What are your thoughts on the risk of exposing proprietary code or data by using the major online tools? Listeners since episode one, thanks for everything you and Leo do . So Randy's right. I did gloss over those risks, so I'm glad he brought it all up. And it's not at all that I meant to downplay them. Given everything we know about cloud breaches, network data interception and decryption and so on. Even if the LLM provider did nothing wrong and made no mistakes, shipping highly valuable source code outside of a company's perimeter creates some risk. However, that said, how many firms are already doing just that by using GitHub? You know, I think that's insane myself, yet it has become common practice to use GitHub for a highly proprietary source code management. I'm not doing that, so perhaps my view is skewed, but it does mean that the company's crown jewels are already exposed outside. Randy corre ctly notes that sending the code up into the cloud for an LLM to rummage around in poses another level of danger. No question about it. And so I completely agree with that in principle . So the use of local models, which I have absolutely no doubt we will someda y see much more in the future, makes a great deal of sense once they become as capable as what's available in the cloud. And at this point, Leo, I heard you mentioning I didn't realize that that there are now laptops being sold without RAM because memory has become so expensive. It's like get your own RAM. Here's what you can plug it into. I I think maybe some people think some some companies think, well, you might have some leftover RAM lying around or whatever, but they just can't get the RAM, so they want to still sell something. Or maybe they think, well, he'll take it from the previous laptop. Exactly. Put it in. Put it in this laptop. Right. Yeah, exactly. Wow. And so anyway, given the insane appetite that data centers have uh for GPUs and things that run AI seems to me it's gonna be a long time before we're able to buy things ourselves that also run AI because we're competing with the data centers that are able to you know purchase all of the next year's production capability. I mean it's crazy. It's crazy. Okay, so um I want to plow now into what Digicert is doing. Uh we got two breaks left. Let's take one now, even though we just did one, and then I will break in the middle of this Digicert uh conversation for our final one. Good. Good. Perfect. This episode brought to you by OutSystems. Speaking about AI, they're the number one AI development platform. But they solve a lot of these little issues that we've been talking about. Out systems helps the businesses uh bridge the enterprise gap to their agenic future. And I know, you know, I think a lot of businesses want to do this, but they're very nervous as they should be about the quality of the code they're going to get, about whether it'll be useful, about whether it's good for just little things or whether you can do big apps. Out systems is here, they've been doing this for 20 years. They know how to do it, they know how to do it for business. They help your business bridge that gap and get to your agentic future where the constraints of the past give way to unlimited capacity and sc ale. Out systems enables your business to build AI agents that actually do work, such as take actions, make decisions, and integrate with data. It's not, it's not a chat bot. It's not just answering questions. It's doing work . And OutSystems provides the only AI development platform that is unified, agile, and enterprise-proven. Let me explain. First of all, it's unified because you build, run, and govern apps and agents on that single out systems platform. It's agile because now you can innovate at the speed of AI, very important, without compromising quality or control . And it is enterprise-proven. They've been trusted by enterprises for mission-critical AI applications and durable innovation. Out systems is the secret weapon behind the world's most successful companies. Not just for those little one-shot apps. They are for massive, complex systems that run banks, insurance companies, and government services. Out systems even helps companies with aging Bridge the gap to the AI future without a rip and replace nightmare. OutSystems provides the safest, fastest way for an enterprise to go from we need an AI strategy to we have a functioning AI application. Stop wondering how AI will change your business and start building the agents that will lead it. Visit outsystems.com/slash twit to see how the world's most innovative enterprises use outsystems to build, deploy, and manage AI apps and agents quickly and cost effectively without compromising reliability and security. And that's fundamental. That's outsystems O-U-T -S-Y-S-T-E-M-S dot com slash T W I T to book a demo. We thank OutSystems so much for supporting Steve and Security now. And you support us when you go to that address, outsystems .com slash twit. Outsystems.com slash twit . All right, let's talk. Okay. The first I learned of some trouble was from someone posting to GRC's Security Now newsgroup with firsthand experience. Oh yep. Peabody, which is his handle , actual name is George, he wrote, This morning, Windows Defender told me it had discovered a severe root kit on my Windows 10 laptop called Win 32 Cardig ant A exclamation point DHA. Now okay, so consider that. Windows Windows Defender tells you you've got a root kit. It's like, what? So you don't take that lightly, right? He says, which it has quarantined. He wrote, searching online tells me this is happening on both Windows 10 and 11 computers worldwide. And one hash involved is that of a legitimate digicert certificate. This is all above my pay grade, but I'm gonna leave things alone for a while and see what happens. Turns out he was right. He said, apparently, lots of people are reinstalling Windows because of this, but I think that's super premature at this point. Right again. He said, My guess is this is a gift from Microsoft, which they will admit to shortly, and if you reformatted your drive, they'll apologize for the inconvenience. With sarcasm there. And unfortunately their apologize their apology left a lot to be desired. Yeah. I was unimpressed. So later that same day, late that this was this c let this past Sunday, bleeping computers, Lawrence Abrams was, all over this and was providing answers. Lawrence's uh Lawrence headlined his news writing Microsoft Defender wrongly flags dig icert certs as Trojan colon win thirty two slash uh sur gent dot a exclamation point DHA . So here's what he wrote. He said Microsoft Defender is detecting legitimate digicert root certificates as, and then that that Trojan name, resulting in widespread false positive alerts, and in some cases, removing certificates from windows. Removing their root certificates, by the way. According to cybersecurity expert Florian Roth, the issue first appe ared after Microsoft added the detections to a defender signature update on April 30th. Today, administrators worldwide began reporting that Digicert root certificate entries were flagged as malware and on affected systems removed from the Windows trust store. Okay, so hold on , to be just to be clear, what a disaster this was . As we know, root certificates anchor the chain of trust for everything that chains down to them with them removed from a system. Nothing that chains down to them will be trusted despite having been trusted just moments before. That's the way the system works, and no one has come up with a better idea for validating signed code. Those two certificates are Digisert's code signing roots. And for example, all of GRCs, my signed apps, are anchored by one of the two of those that were being deleted. So the mistaken removal of those two code signing routes trusted root store automatically and instantly renders every app that was ever signed by a Digicert certificate, you know, who is the, as I said before, the industry's now, now the industry's number one certificate authority renders every one of those invalid and un trusted by Windows . Huge, huge mess . Bleeping Computer continues, writing, these false positives have led to concern. Gee, you think among Windows users, with some thinking their devices were infected and reinstalling the operating system to be safe. Microsoft has reportedly fixed the detections in security intelligence update version 1. 4 49. 4 30. 0 . And the most recent update is now 1.449 nine dot four three one dot zero. Actually I think that should be dot one. Anyway, reports on Reddit indicate that the fix also restores previously removed certificates on affected systems. Well that's nice. So yeah, thank goodness for that. And it's not as if Microsoft had any choice, right, uh about putting them back. It would have been a true disaster if there weren't some immediate means for reverting the specious removal of Digicert's perfectly valid root certificates. As we'll see in a few moments, even though Digicert did suffer a breach, which caused it to misissue a handful of code signing certificates, at no point was the removal of any of their root certificates ever warranted? I mean, that's just nuts. I hope Microsoft will put some safeguards in place to prevent such a thing in the future. Bleeping Computer continues. The new Microsoft Defender updates will automatically install and Windows users can manually force an update by going to Windows Security, Virus and Threat Protection, Protection Updates and clicking on check for updates. After publishing this article, wrote Lawrence, Microsoft confirmed that the false positives were linked to detections for compromised certificates from a recent Digicert breach. Well, linked, but complet ely ridiculous to have deleted the roots . Microsoft told Bleeping Computer. Here it comes. Qu ote. This is Microsoft speaking. Following reports of compromised certificates , Microsoft Defender immediately added detections for mal ware in our Defender Antivirus software to help keep customers protected. Earlier today, we determined false positive alerts were mistakenly triggered and updated the alert logic. Microsoft Defender suppressed and cleaned up the alerts for customer environments. Customers should update to security intelligence version, and then we get that same version number or later, but do not need to take additional action in order, in other words, don't reinstall Windows for these alerts. We've notified affected organizations and recommended administrators look for more details in the service health dashboard, the SHD within the M 36 5 Admin Center, unquote. Huh. Okay, well that's an entirely unsatisfying answer from Microsoft, but I suppose given what Microsoft has become, it's the best we're gonna get and the best we can expect. Nothing they wrote is untrue, but neither should it satisfy anyone who would have appreciated hearing them say something like In response to reports of compromise certificates, Microsoft Defender was a bit overzealous and mistakenly removed some related certificates that should have remained. Whip Microsoft Defender was immediately updated to cure that behavior and has replaced any certificates that were mistakenly deleted . You know, is that so difficult to say? It shouldn't be. Just wait till you see how thoroughly Digicert took full responsibility for their part in this drama. Lawrence's report ing continues, writing The false positives occurred shortly after a disclosed Digicert security incident that enabled threat actors to obtain valid code secur Upon detection, the threat vector was contained. Our subsequent investigation found that the threat actor was able to procure initialization codes, which I'll explain in a sec , for a limited number of code signing certificates, a few of which were used to sign malware. The identified certificates were revoked within 24 hours of discovery, and the revocation date set to their date of issuance. As a precautionary measure, all pending orders within the window of interest were cancelled. Additional details will be provided in our full incident report, unqu So that's a small sample of what good disclosure looks like. Lawrence continues according to Digicert's incident report, attackers targeted the company's support staff meaning Digicert's uh support staff in early April by creating support messages containing a malicious zip disguised as a screenshot. After multiple blocked attempts, one support analysis device was eventually compromised, followed by a second system that went undetected for a time due to an endpoint protection sensor gap. Using access to the breached support environment, the hacker used a feature in Digisuert 's internal support portal that allowed support staff to view customer accounts from the customer's perspective. While limited in scope, this access exposed initialization codes to previously approved but undelivered EV, you know, extended validation code signing certificate or explained Digicert explained possession of an initialization code combined with an approved order is sufficient to obtain the resulting certificate. Since the threat actor was able to obtain these two pieces of information for a finite set of approved orders, they were able to obtain EV code signing certificates across a set of customer accounts and CAs. So great explanation there. Lawrence says Digicert says it revoked 60 60 code signing certificates, including 27 linked to a zong stealer malware campaign. Digicert explained. Eleven were identified in certificate problem reports provided to Digicert by community members linking the certificates to malware, and 16 were identified during our own investigation. This aligns, writes Lawrence, with earlier reports from security researchers who had observed newly issued Digicert EV certificates used in malware campaigns and reported them to Digicert, which of course, you know, that's the nightmare scenario, right? I mean, the re all of the reasons I had to jump through all those hoops in order to get myself an EV certificate. Actually, not even an EV, just a standard validation certificate, is what all of this other mechanism is designed to prevent from happening. Researchers including Squibbly Doo, Malware Hunter Team, and uh Gonks ha reported that certificates issued. I should make you read hacker handles every week. Squibbly do says this? Okay, I'm squib squibbly doo. That's right . Uh issued to well-known companies uh such as Lenovo, Kingston, Shuttle. Now these are the these are the companies to whom these stolen certificates were issued, right ? Lenovo, Kingston, Shuttle Inc. Pallet Microsystems being were all used to sign malware. So the question What do Lenovo, Kingston, Shuttle Inc. and Pallet Microsystems have in common? Posted squibbly do on X . EV certificates from these companies were issued and used by a Chinese crime group, Goldeneye Dog . Uh, and that's an APT known as Q hyphen 27. The malware in this campaign is named Zong Stealer, though analysis indicates it may be more like a remote access Trojan than an info stealer. The researcher says the malware was distributed through the following attacks. Phishing emails deliver a fake image or screenshot, a first-stage executable that displays a decoy image, retrieval of a second stage payload from a cloud storage such as AWS, and the use of, wait for it. Sign bined aries and loaders, including components tied to legitimate vectors or uh vendors. So trusted because signed by Digicert. After Digicert disclosed the incident , the researcher said the incident report explains how the certificates used in these malware campaigns were obtained, because like clearly illegitimate. It should be noted that the certificates flagged by Microsoft Defender are root certificates in the Windows Trust Store and do not match the revoked Digicert code signing certificates used to sign malware. Okay , so that's the great reporting posted Sunday before last by Bleeping Computers founder Lawrence Abrams. Security industry experts have been citing Digicert's upfront incident report as a model of how this should be done. Starting 21 days ago, Digicert began issuing a series of incident reports with each succeeding report updating the previous one, with the final report being posted seven days ago exactly one week ago digicert named this event and the the this final event endpoint two which is that that's the the system where this bad guy was not immediately discovered. And their final report begins with this statement. This is an updated version of our full incident report, which completes the investigation of end point two . Their own overview description differs somewhat from what third parties reported and provide some additional detail. They summarized the whole thing saying on twenty twenty-six four two, so April 2nd, a threat actor contacted Digicert's support team via so a threat actor, right? The bad guy contacted Digisert support team via a customers, a customer chat channel, and delivered a zip file disguised as a customer screenshot. So they were saying, you know, I have a problem. I don't understand how your portal works. Here's a screenshot of the the problem. The file contained a dot SCR, which we know is a screen sav er, executable, containing a malicious payload. CrowdStrike, who is we we know the endpoint security company that does a great job, except that they once brought down all of Windows, but that was you know whoops. Uh uh CrowdStrike and other security measures they wrote successfully blocked four delivery attempts. Caught this guy, said nope, sorry, bad. A fifth attempt compromised endpoint one, a machine used by a support analyst. This delivery attempt was detected and contained by our trust operations team on April 3rd. So five times, five attempts, four were blocked, one got through the next day. It was discovered. Fault they they wrote, following an immediate internal investigation based on the telemetry data at hand, it was assessed that the incident had been contained . Okay, so that's their summary. Then we receive an interesting narrative, some of which Lawrence posted, but the deeper details are interesting. So Digicert said the so this is Digicert , you know, writing it all up. Digicert received the initial third-party report related to this incident on April 5th. Additional third party reports are identified in the timeline. So, okay, just to recap before we go any further, the penetration occurred on April 2nd. Digicert's trust operations team determined it had been con tained the next day on April 3rd. And the first third party report that is coming from some other outside source saying, hey, uh we got some malicious code here that's signed by like a certificate of yours that's fresh. So the first third party report of malicious code found in the wild signed with a then valid Digicert EV code signing certificate occurred the next day, the following day on April 4th. So that's gonna like, whoa, bring Digicert to full atten tion . They report Digicert regularly receives certificate problem reports from community members and security researchers for code signing certificates and proven key compromise cases are revoked pursuant to the code signing baseline requirements. Initial problem reports ultimately linked to this incident report fit within the normal pattern of such revocations. Okay, so they revoked the certificate . Then ten days go by . They write, on April 14 th, further investigation identified that N point two , a different machine, a machine used by another analyst was also compromised through the same delivery vector on April 4th. So that this out of 10 day window. Crowdstrike was not installed on that endpoint, meaning the compromise was not detected during the earlier April 3rd investigation. The machine was established more than three meaning endpoint two that that lit the this newest machine they said was established, meaning you know, set up more than three years ago because our end user machine logs are retained for three years, we cannot determine why CrowdStrike was not installed on this particular endpoint. Okay, so you know at this point it really doesn't matter why. What matters is that because CrowdStrike was not installed on that second infected endpoint two machine, its infected went undetected for 10 days. But in the interest of a full forensic after-the-fact , how did this happen? You know, investigation, they would have liked to know exactly why that machine had apparently never been under the protection of CrowdStrike. Their records only go back three years, and that machine predates that logging cutoff. So today they have no way of knowing what happened back, you know, when that machine was initially brought online, why it didn't get CrowdStrike. And now of course, given that they found one machine that was missing its protection for an unknown reason, the question becomes what other sensitive machines might also be missing their protection? You can imagine that they're going to go find out . Their reporting continues, writing our trust operations investigation found that the threat actor used the compromised analyst endpoint to access Digicert's internal support portal. The threat actor used a limited function within the customer support portal, which allows authenticated digicert support analysts, you know, the the people that we talk to as Digicert customers, I've done that on a number of occasions, to access customer accounts from the customer's perspective to facilitate their support tasks. Makes sense? This access is restricted and does not permit actions such as managing accounts, users' API keys, or submitting or managing orders. However, the threat actor was able to use this function to access probably meaning view initialization codes for orders that were approved , but pending delivery for EV code signing certificate orders across a finite set of customer accounts. They write possession of an initialization code combined with an approved order is sufficient to obtain the resulting certificate. Since the threat actor was able to obtain these two pieces of information for a finite set of approved orders, they were able to obtain EV code signing certificates across a set of customer accounts and CAs. Okay, now so just to put this in context, if you're wondering about Digicert's phrase across a set of customer accounts and CAs . The notion of like why would it be more than one like them, CA ? The notion of differing CAs could seem strange since we're only talking about Digicert. But remember that when I was out shopping around for a new code signing certificate provider earlier this year and finally settled upon Identrust, I discovered that a surprising number of the many apparent alternative certificate authorities all shared utterly identical prices , terms and conditions with each other and with Digicert . It quickly became clear that Digicert had been busily gobbling up much of the competition. So all of these alternative CA s had just become different storefronts for Digicert . And what they've just written confirm s that these various fronts were all sharing Digicert's common back end. I'm not criticizing Digicert. It's, you know, smart business. But it does mean that we now have much reduced competition, and that's not usually best for consumers. Okay, so now we get some statistics and numbers. They write during our investigation between April 14th and 17th as Digicert identified certificates potentially affected by the threat actors' actions. We revoke them. Digicert revoked sixty 6 0 certificates issued from the following CAs. And there's four of them: Digicert, trusted G4, code signing. Uh, and they've got uh a a bunch of different specs on that. Uh the and an another of the same, then uh then one called go get SSL G4 code signing. So that's probably one of the what one of the other compromised sub-CAs and also something called Viro Key High Assurance Secure Code EV . So those were the those were the root CAs that um that had been used to sign those certs. They wrote 27 of the 60 revoked certificates were explicitly linked to the threat actor. Eleven were identified in certific ate problem reports provided to DigisERP by community members linking the certificates to malware, and sixteen were identified during our own investigation. During our investigation, um had included review of the threat actor's activity in the support system, as well as tracing delivery to IP addresses known to have been used by the threat actor. You know, so they they took they they they they got information about the known problem certificates then they looked at the metadata surrounding the issuance of those certificates and then were able to use that to broaden their search and find any other certificates that that threat the same threat actor had also managed to issue to itself. And so they were able to say the IP addresses used by the bad actor to install certificates included, and they provided in their report unredacted IPs, you know, 82.23. 186.8 and so on. There's a bunch of them there. So those are the IPs that that that the bad guy used in order to compromise digicert. They said in addition to the twenty seven fraudulently issue and revoked certificates identified above, thirty-three of the total sixty certificates were revoked during our own investigation as a precautionary measure. For these certificates, we could not explicitly confirm customer control. In addition, pending orders were canceled, closing access to the threat actor. All identified certificates were revoked within 24 hours of discovery with the revocation date set to their date of issuance. So, you know, note that we keep seeing this language within 24 hours of discovery. Uh, this is Digicert explicitly asserting that it has carefully followed the well-established CA browser forum guidelines for proper CA behavior. This is where as previously as previously we've seen and reported, the other disgraced certificate authorities f fell well short. You know, those others were you know under the rug sweepers, you know, first hoping that no one would notice and catch them in their mistakes, and then once they'd been exposed, you know, they worked overtime to minimize and hide their failures. Uh, you know, and the you know the truth is no one expects anyone in this arena to be perfect. Perfection is not a requirement. Proper behavior and acknowledgement of a f of a mistake, that's the requirement. So uh that's what Digisert is busy doing here . Uh they wrap up their initial overview by writing the exploited certificates identified by the community member were found to have been issued by and to to sign the zong stealer malware family and so forth and basically the same stuff that Lawrence talked about. So the really interesting stuff comes next, uh, it being how they take themselves to task over how this happened and in detail the contributing factors that facilitated the attacker's success. So, what I'm about to share is the reason I gave today's podcast the title Digisert Doing It Right. This is written so objectively that it feels more like the work of outside auditors than Digicert's own staff. It's just so difficult to fully disconnect one's own ego from you know truly self-indicting statements. But to digicert's credit, there was no sign of that, you know, the typical rolling disclosure that we've seen so many times elsewhere. You know, Microsoft, for one, could certainly learn a two, a thing or three from Digicert . So we're gonna talk next about their headline root cause analysis, Leo, after we take our final break. Okie dokie . Man . We live in a dangerous world. Well we yeah . We we we live in a world where a huge amount of industry is being applied by the bad guys. Yeah. I mean, remember how at the beginning of this podcast we had cute little viruses and they they used to infect people and we go oh look at that what what does it do? Nothing it just propagates. Why? Well because, it can. But it makes it defaces your web page. That's all. Everything changed. Everything changed when cryptocurrency allowed people allowed the bad guys to get paid. Yep. It turned it into a business model. Well, fortunately, there's ThreatLocker, our sponsor for this segment of security now. Little zero trust goes a long way. ThreatLocker's Zero Trust Platform delivers the industry's most comprehensive suite of zero trust solutions. They just added protection for networks and the cloud. By extending Zero Trust enforcement to cloud services and company networks, I mean, this is huge. Threatlocker ensures that devices are validated through a secure broker before connecting to platforms like the ones you're using, Salesforce or Microsoft 365, Asana, Google Workspace, GitHub. This means even if a user is successfully fished, and this is $64,000 question, right? Can you keep your users from getting fished? But even if a user is successfully fished , with this zero trust platform, attackers cannot access resources. It stops them. They'd have to have physical possessions of the user's trusted device and authenticate through that . I mean, that's that's a lot better, isn't it? Threadlocker works across all industries. Uh they have fantastic 24-7 US-based support. I've talked to them. I know how good they are and how much they care . It works in every environment, Windows, Mac, Linux, absolutely. It enables comprehensive visibility and control. And I'll tell you, when you look at the uh the control panel and you see how ThreatLocker is doing it. In fact, one of the things that's worth doing, if nothing else, get a demo because you'll see right there on the control panel who's trying to access your system and what they're using. You know, you may not know that you have 16 different remote access tools running on your company network, but you you put threat locker on there, you'll know immediately. So it's just great for diagnosis alone. Uh, but of course it does a whole lot more. Just ask Rob Thackeray. He's the end user technical architect at Heathrow Airport . You know, Heathrow's had problems in the past. I think there was a real burden on Rob to make sure that Heathrow was absolutely locked down. That's why they chose ThreatLocker. He said, and this is a direct quote, quote, ThreatLocker was the most intuitive solution we tested, and the responsiveness of the organization , the willingness to engage with us, to set up a demo, to work with us on weekly audit reviews was very good. It is great to have an ongoing relationship with a company that's so responsive to our requests. Thank you, Rob. Threatlock is trusted by global enterprises that just can't afford to be down. They cannot afford to be ransomware. They cannot afford to be fished. Companies like JetBlue. They use ThreatLocker, the Indianapolis Colts, the Port of Vancouver. ThreatLocker consistently receives high honors and industry recognition, a G2 high performer and best support for enterprise summer 2025. Peer spot, ranked threatLocker number one in application control . Get app gave ThreatLocker their best functionality and features award in twenty twenty five. And I can go on. I I won't, but I mean they've won all the awards. You confidently ensure that users have access to a consistent, safe network connection. Offices, remote users, internal servers, and critical services can maintain smooth operations without the need to open inbound ports. You don't even have to deploy traditional VPN solutions. Your end users will get the secure, reliable internal system access they need wherever they are without complex infrastructure changes. It really works. Get unprecedented protection quickly, easily, and cost effectively with ThreatLocker. Do this for yourself. Visit ThreatLocker.com slash twit. Get a free thirty day trial. Learn more about how ThreatLocker can help mitigate unknown threats and ensure compliance. That's threatlocker. com slash twit. Threatlocker.com slash twit. We thank them so much for their support of security now . And now back to the uh Digicert story with Steve. So under technical background, they said understanding this incident requires understanding how EV code signing certificates are issued on hardware tokens. The c ustomer requests a code signing certificate from Digicert . Following validation, Digicert securely provides an init what they call an initialization code. We've heard that term throughout this. An initial an initialization code to the customer. The customer installs or already has installed Digicert's hardware certificate installer software locally, meaning at their end. The customer inputs the initialization code into the installer, which generates key pairs on the hardware token and submits the public key to the CA . The CA generates the certificates against the approved order. The installer retrieves the resulting certificate and installs it on the token. They said the process is described in a public knowledge base, and they provide the link. Possession of the initialization code combined with an approved order is functionally sufficient to generate and retrieve the corresponding certificate. The initialization code operates as a bearer credenti al for the approved order and is single use. This feature made it apparent which initial code initialization codes have been used. Okay, so I I've done exactly this in prior years with Digicert. And when you think about it, the process of issuing a certificate that like creating and issuing, you know, creating and signing uh that will be contained in a on a hardware token is a little trickier than you might just imagine at first. The need is for the priv ate key half of the public-private key p air to never, ever, for any reason, ever , just to be clear, ever exist. Never thank you, Leo. Outside the hardware. It it's in the key and it never leaves. That means that it must be generated by the dongle inside the dongle, and that the dongle will never export its ultra-protected hardware key . So web server public private key pairs have no such requirement. So a web server just uses the underlying operating system's cryptographic system to synthesize a key pair, a public key pair for it. It holds onto the private key while the public key is placed into a CSR, a certificate signing request, which the CA signs and returns. But forcing the private key to never leave the hardware dongle . Very much complicates matters. The point here is that Digicert issues these initialization codes against a customer's account sends them to the customer who then uses them with Digicert's own hardware certificate installer app running on the client's machine with the hardware dongle plugged into it , the code validates, you know, the code which is ingested by their app, which they provide, validates their right to have a certificate by communicating on the back end with Digicert's APIs and servers , then it triggers the key pair generation on the hardware dongle, the public then then the hardware dongle uh does allow the public side the public key to be sent out which the the app then uploads to Digicert and the install uh and the installation of the signed certificate. So Digicert then signs that public key, sends it back to the app, which then installs it back into the hardware . So a lot is going on that the user never sees from the user's perspective, it just kind of is magic. You you enter your code and then you say go baby, and a minute or two later it says, Okay, you got your key. You know, you're all set to go. But the you know, the point is that the stateful nature of the operation creates some points of exploitability. There exists a window from the time that initialization code is issued to the time it is that it it is actually applied where the bad guys who are able to get it could install it in their own hardware rather than the customer installing it in their hardware. And these are big companies, right? Uh Lenovo, for example, where they're you know, they've got teams doing things, they probably and they have got initialization codes that are just have been issued but haven't been used yet. And so the bad guys took advantage of that window. So Digicert states four what they called contributing factors, things that things about the way their system is and uh actually now was that helped this to happen contributing factor one they said inconsistent or incomplete endpoint detection coverage. Well we know that right they said security tooling crowd strike was not uniformly configured by Digicert across the user popul ation exposed to the attack. The crowd strike prevention setting on N point one was below the intended organiz ational standard at the time of the initial compromise, allowing the malicious payload to execute before blocking eng aged. The CrowdStrike sensor on endpoint two was not installed. As a result, no detection fired on the compromise machine. Logs for end entity machines are retained for three years. Since this machine was set up more than three years ago, our security team could not determine why this particular machine either did not have uh Did not have an installed crowd strike sensor . They also said the sensor not being installed was identified on April 14th during the expanded investigation triggered by the third -party report. You know, they thought they had it contained on the fourth. Ten days later, it's like, oh crap, something's bad big and bad has happened. They said the original investigation on April 4th did not include a check of EDR, you know, endpoint detection enrollment status for all exposed users. If the sensor had been installed on endpoint two , the connection on endpoint two would likely have been detected and contained in the same time frame as the other targeted machine. That was M.1. This created the window during which the threat actor was able to access the portal function. And then we're about to talk about that in the next contributing factor. And harvest initialization codes, which actually is the third contributing factor. Okay. So the second contributing factor, insufficient privilege minimization in the support portal function. Again, taking full responsibility for how the bad guy was able to get up to as much as they were. Insufficient privilege. So minimization. So they wrote Digicert's internal support portal includes a function that allows authenticated support analysts to proxy into specific customer accounts to facilitate customer support. You know, like viewing it the way the customer does. I don't really understand what's going on. Can you show me what I'm supposed to be seeing? And so the support guy says, okay, let me get onto your account and goes, ah, I see what you mean. So they wrote, in this mode, certain functions are masked from the analyst. However, access to initialization codes for pending code signing certificate orders was not among the masked data elements. And they're saying it could have been. Leaving those codes accessible to support uh an analyst operating in a proxied session . They said the portal function had not The portal function had not been formally classified within Digisert's privileged access management PAM framework. The definition of privileged access was primarily scoped to direct access to CA and did not encompass this indirect account management function that had a path to certificate issuance. As a result, the portal function was not subject to the PAM controls applicable to privileged users under the CAB net security including formal threat modeling against misuse scenarios, least privilege design review and access recertification. In other words, in other words, they missed this. And they recognize that this did not have to be. They said the portal function is a long-standing feature. On April 14th and 15th, following the discovery of the incident, we deployed a code change to mask initialization codes from proxied users on both our US and EU platforms using either the UI or the API. The absence of this initialization code mask ing was identified during the investigation triggered by the third party report on April 14th. And finally, interaction with other factors. This factor, you know, this second contributing factor defines the scope of the damage enabled by the contributing factor number one, which was the lack of endpoint coverage. Without the EDR gap, the dwell time would have been minimal and the number of initialization codes accessible would have been limited. This factor also interacts with contributing factor three as the absence of masking is a direct consequence of the codes not being recognized as requiring credential tier protection. Which brings us to contributing factor number three. Initial c initialization codes not protected as bearer credentials, meaning the initialization codes did not were not considered to have sufficient need for protection. They wrote the EV code signing certificate pickup workflow was designed with a threat model that assumed initialization codes would only be accessible to the validated subscriber delivered via a secure channel and entered by the subscriber into their local hardware certificate installer. The threat model did not account for the scenario in which initialization codes stored within Digicert's internal support portal could be viewed by a compromised Digicert analyst account operating through the portal function. Again, whoopsie. Therefore , initialization codes were classified as intermediate workflow data rather than bearer credentials, which would have elevated their need for protection to a higher level. This initialization code workflow, they wrote, was designed at the time the EV code signing token issuance process was developed. The issue was identified during the investigation triggered by the third party report on April 14 th was not identified through any previous design review prior to this incident. So this just kept getting missed because again, you can look at things, but you sometimes you just don't see them. And what and their point is that this mistaken definition allowed it never to be properly classified and then it did not automatically fall within the proper security context and constraint. So everything we're seeing here is the work, and it should sound like it, of true forensic security profession als patiently working to understand step by step not only what happened, but why a supposedly carefully designed security system that was even designed from a theoretical premise set of theoretical premises of what we want . Even so, how this was how it that allowed this to happen. So the result is guidance about what can be changed to prevent successful exploitation at each stage. And this brings us to the fourth and final contributing factor, which explains how the bad guys managed to infect two to infect Digisearch two analysts in the first place. They wrote overly permissive file transfer capability in c ustomer fac ing support channels. They said Digisearch customer support chat channel and Salesforce case attachment workflow permitted delivery of inbound file attachments from external parties, including the general public, to CA support staff with insufficient restrictions on file type , automated sandboxing, and content ins pection. This created a direct delivery path for malicious executable content to personnel having privileged access. The support chat channel had not been adequately evaluated as a potential attack surface for malware delivery against certificate authority staff. The support chat channel and Salesforce case attachment integration were operational prior to this incident. The delivery vector was confirmed on April 4th and 5th, at which point additional malicious zips were identi fied across other sales force cases and removed, which is significant, right? There were other infections that hadn't had a chance to like take hold. Corrective controls, file type restrictions, sandboxing evaluation are in progress as described in action items. The number of channels by which customers can re ach support staff has grown. The file controls on the chat channel were believed to be sufficient prior to this incident. This factor is the initial attack vector that enabled all subsequent factors to come into play . And this brings us to the lessons learned, conclusion, you know, with what went well and what did not go well and where they got lucky. Um so um they explained under what did go well rapid initial containment on endpoint machines where EDR was working as intended. For these machines, DigisErt's trust operations team completed containment, process termination, registry cleanup, and artifact deletion within hours of detection on April 3rd. Also, what went well? Proactive identification of the full delivery chain. They were able forensically to catch that. The investigation identified the Salesforce case attachment autoconversion mechanism as the delivery path and located additional malicious zip files across other cases before they could be opened, preventing further compromises from the same campaign. So when they talked about earlier, they did not throw Salesforce under the bus, but they talked about how additional points of entry have been added. Apparently it was the incorporation of the Salesforce uh services that allowed this stuff to get in that way. And it sort of snuck by. They said also what went well: same-day remediation of contributing vulnerabilities during incident response, critical fix es were implemented without deferral. Crowd strike prevention settings corrected on April 4th. OctaFast pass disabled and multi-factor authentication tightened on April 14th. Initialization code mask ing deployed across US and EU environments on the 14th and 15th. And confident attribution of the second compromise through forensic analysis. They did figure out exactly what happened after, you know, with that N point two machine. They said linking across compromise machines, activity logs to the same threat actor required an analysts across identity analysis, across identity events, endpoint telemetry, and support workflow artifacts. So if that all went well, what did not? File type controls turned out to be insufficient on customer facing support channels. Inconsistent and incomplete EDR coverage enabled a blind spot that was unknown prior to the incident and that directly enabled the attacker's dwell time, you know, giving them 10 days. Initialization codes were not protected as bearer credentials as they should have been. The portal function exposed initialization c odes that are functionally equivalent to the certificates they enable, because they were classified as intermediate workflow data rather than credential material requiring masking and credential tier protections. Also, what did not go well, device bound authentication. OctaFastPass was used as an MFA, a multifactor authentication bypass. Fast pass allowed a threat actor operating on a compromised device to inherit the device's authenticated session and satisfy multi-factor authentication requirements without a genuine second factor, enabling access to the portal initialization codes after the initial endpoint compromise. And finally, following endpoint uh containment that M point one containment on April third, the investigation concluded, you know, obviously incorrectly, that compromise attempts had been neutralized without validating all endpoints exposed through the same delivery vector. So somebody a little too too quickly said, okay, we're done here . In retrospect, of course, I imagine that they now wish that after that first attack, which was thwarted by CrowdStrike's endpoint defense after a brief window, that they had checked for other instances of that malicious zip file across the rest of their network. We now know that they did that later and did find a handful of other instances of it. So it's sort of unclear why that did not happen at the time. And I imagine somebody's asking somebody. So they finally share two points in conclusion where they felt they got lucky. They said a community member involved in security research reported the evolving pattern of misused certificates and engaged in dialogue with our support with our support team. Without that report, the undetected compromise of N point two and the associated misissu ance might have remained undiscovered for a longer period. Nice of them to admit that. Our investigation also they got lucky where our investigation indicates that the threat actors' activity was focused on gaining access to code signing certificates. A differently motivated threat actor might have attempted to use the compromised account for broader actions. Several of our action items are designed to address this risk. And then they conclude with a final three points. This incident demonstrates that internal support tooling with indirect paths to certificate issuance must be subject to the same security scrutiny as their certificate authority issuing infrastructure. Tools that were designed for legitimate operational purposes can become high value attack targets. Second, the incident also illustrates the importance of defin ing privileged access broadly enough to encompass any system or function with a path to certificate issuance, not just those with direct access to HSM's har,dware security mod ules, or signing infrastructure. And finally, the dwell time underscores the importance of comprehensive post-incident investigation scope, meaning don't quit your investigation prematurely and continuous EDR coverage monitoring, like make sure all your endpoints are actually being monitored. A single missed endpoint can negate the value of rapid containment elsewhere. So, you know, they they go on to actually enumerate 21 individual action items, many of which they already articulated or implied. So I'm not gonna take us thankfully through each of those. Suffice to say that there's little doubt that Digicert was extremely unhappy and embarrassed by this event, right? Everything any certificate authority is about, and a huge amount of security focused design and third party contractor support from the likes of CrowdStrike and several others, all of this was intended to prevent anything like this from ever happening. But it did anyway . But at no point did we see what we've previously observed from so many other certificate authorities who have been struck by similar abuse, or even by Microsoft, who you know has nothing to lose. No one's gonna leave Windows. Almost invariably they first hoped, you know, the other guys, these these these really, you know, denial certificate authorities first hoped that nobody would notice. Then when someone did , they would downplay the severity of the incident, hoping that that would be it. Then when additional evidence of further exploit came to light and surfaced, oh, they'd apologize with some lame excuse about having intended to mention that too. Uh-huh. Yeah, right. Well, we have from Digicert, the industry's largest commercial certificate authority, second only to let's encrypt , which you know is free, is full public disclosure and responsibility taking. The result has been a deep analysis followed by true action to prevent like at multi-st multiple levels and stages, not just like closing the front door, but all the hallways in between, you know, to prevent anything like this from transpiring again. And as usual, uh I think that's more to recommend them than not. If their code signing certificates were not so unreasonably expensive, I would not have invested so much time earlier this year preparing to jump ship and find another supplier. I'm glad I'll be moving to IdenTrust, but I'm only doing so because I object on principle to, you know, unconscionable costs for certificates, not the quality of their work . But as for Microsoft , given what we now know , it is unbelievably difficult to under stand, to explain, to explain away , to excuse how Microsoft could have possibly fumbled their end of this so thoroughly. There would presumably have been some dialogue between Microsoft and Digicert, where Digicert provided Microsoft with the thumbprints or serial numbers of the 6D certificates. They had revoked and blacklisted so that Microsoft could do the same with Mike with Windows Defender. You know, as we know, revocation is an imperfect answer with certificate management , but it's all we've got. So Microsoft would have definitely wanted to add those certificates, those 60 certificates, to Windows Defender's existing code signing deny list, so that nothing signing them would have been allowed to run on Windows. But that just entails checking thumbprints against defenders deny list . How Microsoft could possibly have fumbled this into the removal of Digicert route certificates for all certificates ever issued in the world ever is just impossible to understand. It feels very much as though someone in control of that important process doesn't know what they're doing, which is horrifying. I hope AI AI didn't do it. Uh anyway, fortunately, that too was fixable and it was quickly fixed. So we go on, Leo, to the next adventure. I love it that Microsoft kind of the the way they phrase this sounds like they're blaming Defender. They say in response to reports of compromise certificates, Microsoft Defender was a bit overzealous . Well, it it certainly acted poorly So this is interesting. So the i they were initially compromised through s customer support chat. Files uploaded through chat. Maybe chat or it sounds like maybe s they've added some Salesforce software as a service thing. Right. Because there was a mention of Salesforce and they may that may be the chat. That the ba exactly it might be Salesforce chat. And so so that it did get in that way. So so some somebody said to to the support guy, hey, I don't understand what I'm seeing here. Let me send you a screenshot. Yeah. Here's yeah. Or here's a zip file of something. Yeah. And then the the CSR, who had escalated privileges, which is a problem, unzipped it. It attacked. They responded very quickly. But this showed, this is the issue uh with certificates, unlike say a ransomware attack, although that can happen pretty quickly. But if I have 10 minutes with Digicert and I can get their root certificates, I'm golden. I don't need more than that. Or time than that. Yeah. Wow, and I mean it is it uh one of the things that I did think was that we clearly have a system, we have so many security firms that are that are looking for malware when they talked about, you know, an industry partner or a third party, you know, it was some security firm who who they didn't identify that that called up and said, Hey, uh, we're seeing some malware that was signed an hour ago by Lenovo and it's your cert. So what do you think about that? And then so I mean so the the we have you know there's a lot of closed loops here. Oh yeah that are it's good that they're closed and that they're looping. There's really a hu a huge kind of web of uh threat analysis. It's a big ecosystem. It's a really big ecosystem and and they know that they have to work hand in hand. They have to work together. So in some ways, you know, that's been the response to all of these uh attacks is is a a much improved, I think, uh early warning system. Which which, you know, here I am trying to publish my little software and being blackballed by Windows Defender. You you'd think that with this system working as well as it is, they could wait until actually seeing malware and then then you know you know ru um you know call digicert to report me but no defender says i don't know about this they removed the entire brain instead of just the instead of just the tumor. They just said, well, just take the whole brain out. You don't need that, do you? Yeah, we've decided we're not gonna tr trust anything Digicert has ever signed. I mean it's unbelievable. Yeah, that's a very bad response. Uh it's almost I you know it feels like a panic response. Like they were so freaked out that they overreacted. And it wasn't Defender doing this, it was some human or maybe some AI. I don't know. I don't think's it like it sounds like somebody really to to to go in and remove a route. You had to know you were removing not at no point did the response to Digicert require removing anything. It meant checking thumb prints. Does this thumbprint match? That's all. So how it got extended, I mean, again, I do actually wonder, I had the thought before I was recording this with you, Leo, could this have been AI that di that that hallucinated a root cert removal? Is that what we're now doing that what we're now gonna be dealing with ? If so get your seatbelt. Yeah. Buckle up, baby. Yeah. Well, what a good story though. And I'm glad uh Digistert did the right thing. Oh, I I again I just uh you know, a hat tip to those guys, the fact that they ruthlessly not only s you know investigated, but then self-reported. Yeah. I mean you no one could ever take issue with with the way they're behaving. I'm sure that was their goal, too, right? I mean, they they don't want anyone to have any doubts about them. And who would after this? I would trust them more now. Right . Good stuff. Thank you, Steve. Steve Gibson uh does this every Tuesday. As you probably know, I'm sure you make this a regular stop on your uh podcast list. You can uh catch the show live if you want to get it right away. We we stream it live as we're doing it Tuesdays right after Mac Break Weekly. That's 130 Pacific, 430 Eastern, 2030 UTC. The live streams, of course, in the club twit Discord, but also YouTube , Twitch, X, Facebook, LinkedIn, and Kick. So watch wherever you want. Um, if you aren't around on a Tuesday morning, then or afternoon or evening, depending on where you are, uh, we always it's a podcast. You can always get copies of the show. Steve's got uh actually completely unique versions of it . So depending on what version you want, you would it's the same show, but he's got the 16 kilobit audio versions, which look admittedly not the highest fidelity, but very small. 64 kilobit, which is full fidelity. He's got the show notes, which are amazing. And uh and uh I think a lot of people like to read along as they're listening. Well, it's also a great reference uh to have. Uh he's very complete in the show notes, 22 pages today of show notes. You'll get those uh either via email or directly from his website, gr c.com. He also has transcripts. Take a couple of days because they're written by a human. They go up a couple of days after the show , so that's a good way to search. Uh or again to read along as you uh listen. All of that at grc.com. If you do want to get the email transcripts, go to grc.com slash email . And there's there's a the initial point of this page was to whitelist your email so you can send Steve questions, comments, suggestions, picks for the picture of the week, that kind of thing. Uh so there's an email form and he'll validate your email, but below it , since you're providing the email, there's also two checkboxes, one for the weekly newsletter that contains all of the show notes, the other for a very infrequent uh email when he's got a new product, like of course, the world's best mass storage, maintenance, and recovery utility, SpinWrite, currently 6.1, and uh his DNS Benchmark Pro, which just came out a few months ago. Uh that is uh uh nine dollars and ninety-nine cents and very much worth it. Uh I you know, frequently we say, Oh, the internet's slow today, and it's not really the internet that's slow, it's your DNS server that's slow. Something's going on with Quad Nine, by the way. If anybody's using quad nine, I I've noticed it's its cache response is very fast, but it's when when it when the quad nine server doesn't have something in its cache, it needs to go out and it is really, it's like taking half a second for it to get a response back. So And that's something DNS Benchmark Pro would let you know. That's how I found out about it. Yeah, it would also give you good alternatives, right? If you, you know, that are faster. Um, so grc.com, the Gibson Research Corporation. You can go to our website, twit.tv slash sn for the 128 kilobit audio, which is, to be honest, not any better than the 64 kilobit audio, but we need to do it because Apple down samples and they need a high quality one to downsample it. It's a long story. We also have video, which Steve does not have, uh, and that is at twit.tv slash SN. There's also a YouTube channel for the video, great way to share clips. I think a lot of times people want little clips of pieces of this to send to the boss or your IT team or friends, whatever. Uh, that's the easiest way to do it. Everybody can watch YouTube. But the easiest way to make sure you don't miss an episode is to sub subscribe in your favorite podcast client. That way you'll get automatically uh as soon as we've polished it all up. Ben ito uh be working on that this afternoon. Uh Steve, thank you so much. Great show. And we'll see you next week on Security Now. Episode 1079 coming up. Hi there, Leo Laporte here. I just wanted to let you know about some of the other shows we do on this network. You probably already know about this week on tech. Every Sunday I bring together some of the top journalists in the tech field to talk about the tech stories. It's a wonderful chance for you to keep up on what's going on with tech, plus be entertained by some very bright and fun lines. I hope you'll tune in every Sunday for This Week in Tech. Just go to your favorite podcast client and subscribe, This Week in Tech from the TWIT Network. Thank you .