Shortcuts Playground Project Launch

Hello and welcome to episode six hundred and four of Connected. worldlds podcast My name is Steven Hacket. I'm your annual chairman And I'm here to introduce Mike. Hi, Mike. Oh, hello. I'm so happy to be back. Yes, welcome back. How was your time away It was good. It was good. I have a very quick funny story to tell you. I will just introduce the keynote chairman Vover Ch Chi Cho, Veterico. hello Cho Mike. How are you? Very good, very good. So I was listening to the connected podcast. I had already been told that there was a segment I wasn't supposed to listen to. Actually, before I was told that, I saw the chapter names and it said, Mike don't listen to this chapter, which I thought was very funny a chapter name purely for one person. Yeah is which is pretty good. like everybody else got that You are more than this person, you are a C be anost Cost Okay. But so I knew this was coming, right? And I was taking a walk to a go pick upp coffee in the morning. We were in Romania. It's the first trip overseas with a baby. went really well actually We're happy about that And I was walking down the street cararrying two flat lights. O in each hand And I listening to the podcast and I'm like, okay, Mike, stop listening now. Mike Ohh God I didn't have any free hands. put I put a coffee down on the street on the floor like the pavement on the street to to be the floor of the street to the floor of the street as we call it. That's how it's called in Romania. they call it the floor of the street. It's like a whole thing U And then I so yeah, I did put put the coffee down in the in the street on the street floor and so I could pause it and market is done This is fantastic. Thank you for not cheating or listening or whatever. I can't even tell you what it's about. I mean, I know I know what it was even though I haven't listened. Yeah there is precedent for this. Like I' aware of what probably occurred. Did't Yeah, it's not it's not time. It's not time yet. Okaykay It's not time. I think it's July. So you know when it is. Okay. Yeah, I I Last week looked because I figured you were going to do this to me U Are we that predictable Federica? A are we that predictable to him? You told me don't listen to this chapter. It's happened in two other instances of which I've been a part of the creation of those episodes. So like it was pretty obvious what was happening. That's fair. And yeah, and I think it's sometime in July that we need to resolve the hack Olympics. Yeah, um We got a bunch of stuff to talk about today It's Google IO season. So just everybody Everybody buckle in for that But first We have to talk about some things we've already talked about and follow up Last week Federico and I spoke about what is happening. You were just like typing nonsense into the outline Um, We were talking about The the pros of edable user interfaces Right somethingomething that I think all three of us are in favor of And Edward Munn wrote in Omastodon And I cannot believe Like, why isn't we talking about this? I can't believe we didn't talk about this Edward Wites The iPod app on the very first iPhone had edit button to customize the tab bar. What a shame this wasn't preserved. So Remember this. And if you go back and watch the introduction of the iPhone like the iPod touch, that's in here I think the default was playlist artists songs and albums. But if you say that you didn't care about songs and you wanted podcasts in there or audiob books, You could change What was in the tab bar and then the rest of it was just in the overflow menu and You know, not very many IOS apps let you address the tab bar. We mentioned a couple like likeike Iivory But u Apple had it as a first party thing back in the day And it was good. Yeah, I mean, what it was is no matter what happened, you just had to get podcasts in there, right? It's like you got to get rid of something to get podcasts in there. And that's what I would do and I put it in. Yeah. This is another remnant of the like infection of You two Apple, right? So we have The iPod which was ridiculous that they just did one special edition iPod and it was the YouTube iPod. We had songs of Innocence And then also the artists icon. In the music app was was the outline of Bono It's it's deep It's weird. It is weird. It's weird. Why this band, you know? Yeah. this is one band that's everywhere. They's just like no one else got this treatment. That's true That's true. L little bit about spotting the International Space Station. Jonas wrote in to point us to a web app that they built that allows you to check when the space station was historically closest to a location not just visible from it. So really like when it is directly overhead, B the link to this will be in the show notes. It is very cleverly called astronauts for Dinner, which I think is a very good name for this Um And you can plug in an address or coordinates. and seeee the last time and all the previous times, the International Space Station was overhead So this was in another anecdote of me listening to the show last week as a listener moment when you said about, you know I used to I like to know when the International Space Station is overhead. I also had the same feelings as Federico of Why?ike what are you gonna do with that? Like what do you do with that information you saw it did you know? Yeah. So I just plugged in one infinite loop And the closest it ever got was february eighth, nineteen ninety nine Zoomed right over as they were developing OS ten Why can I plug in let me plug in my location. Wyn't you want to read your location out loud while you type it? No put in by current location at the studio zoom in Loading nearby passes. Yep Oh man, the last one was august fourteenth, two thousand six. So that I guess this is the reason of like why is good this all Oh no I just missed it. seventeenth No they're not sorted properly. Act they're sorted, I think by the by the distance away specific closeness What does it mean four meters then? It was not four meters away from my apartment. I think like longitude and latitude wise Yeah This is like it's like L I didn't write this website just in Push if I open the window, there's the international space station here guys You can give a national Iive five far as managed to get. All right, it was fifty meters away in january twenty twenty four. Pretty close. Pretty close. I missed one like basically exactly overhead in November. It was like three meters off. And if you had H Assistant running, you could have gotten a notification. Yeah, but I probably was going to be at home So You could have businessed so. Oh, that's right. you put your shoeo. Yeah, addresson here Um Very clever. like I like this website. I think it's fun Thank you, Jonas Thank you, Donas king about well what you do when it's going to f of her head? Jay wrote in u and shared a photo that they took, actually six photos they took of the International Space Station transiting the sky U And so they were able to stack them. I think this is in front of the sun I guess it is looks like the sun It't look like the moon Um And you can see it That's the orange moon we have That'ange. I think the sun is the orange one, but don't quote me on that. Okay Oh, I've just accidentally added this photo to my photo library. Nice. Sure I don't think that's what I want to do, is it? think I think that's fine It's an iCloud link. I think we need to put that somewhere else if we're going to put it in the show. Okay. yeah, I'll put it on Wh is us we're just like, Oh we kind of like songs of innocence this photo into everybody's photo Well, it probably the iCloud link probably doesn't last forever, right? I don't I don't know whoo could tell? I mean, the funny thing is now I can't find this image. So that's just in there. It's just buried away in there somewhere. Yeah, you gotta go to Reons to find it. othertherwise in the library it's like Like it has a date metadata. twenty twenty four It's so hard to find recents on the Mac Like it's really easy on the on the iPhone, but I find it difficult to find like recents on the Well I'll have a link to some place you can see that Federic, you talk about this double tap bottom of the screen seri business Yeah, Mike Rodin and asked interesteresting takes on the new incoming series. Do you use the double tap bottom of screen to type to Siri? I wonder how much of the app will be similar to that. Almost sounds from your takes, it will be moving to the top of the screen instead of the bottom. I try to make myself use. I try to make myself use it and started a little bit of a habit now. Your takes just reminded me of that feature So the tyype to seri graduated from an accessibility feature to a proper system feature Uh A couple of years ago, like a year and a half ago, barely use it. like Ver few instances on my end of wanting to do something with Siri but not being able to speak to Siri And so in those cases, I actually did a double tap on the home indicator to type to S instead U I think So there's a couple of problems here Obviously, the fact that Siri isn't great doesn't exactly incentivize the use of types to Siri And second, the activation method itself. I think From the description that German gave, I think it's gonna be much easier to just pull down from the dynamic island you know, kind of kind of do it very quickly than it is to precisely tap on the home indicator. Yeah. So the gesture sounds easier in theory I totally forgot that that double tap thing even existed But if Ciri will be smarter, then you will also be more incentivized to actually use it in a variety of cases. I think for me there is an issue in haaving two different ways to invoke Syria no matter what they are Like where I feel like I would be happier to, you know, long press the the sleep wake button and be able to then start typing or talking. You know, like maybe I tap another on screen button The keyboard or somet time There is like a The cognitive load of like, I want to talk to Siri Okay, Do I want to type or speak? and then choose the activation method That is For that method, you like for that way of interacting I just feel like you just have one of them. Just not me long press it If the screen's open and I'm looking at it, maybe just bring up a keyboard as well and I can speak into it or I can type into it. Why do we need to have two different ways to talk to Siri? Like it doesn't too me, that doesn't make any sense, really? I get it. I get it. Dn't government say it the new. design and the dynamic Aland will feature both input modes like at the same time. So in theory you should be able to switch Because I can imagine like maybe you start with a chat query and then you realize, no, actually I want to speak and you press a microphone button Maybe they could do that. I don't know. Yeah. Do you think if they did that they would get rid of the the lung press on the bt No, I don't think they will. I think they will get rid of the double tap on the home thing. Yeah. not the side button though We don't need that bar anymore, right? No, I don't like we do. We can get rid of. Isn't isn't it it's also kind of weird now, I think, because I think as OioS twenty six, the bar gets hidden by default in most apps. So like if you want to use type two Si now, you have to double tap an invisible space Maybe this maybe that is part of them like starting to edgeuc to phase it out. It doesn't exist anymore. because it's kind of wasted space on the phone Like I don't I don't think that a new user to an iPhone is helped by that being there. Anyway. Like I think it helped a lot when we were doing the transition, right from buttons to just screens But I think at this point, if you're learning the iPhone I don't think it would be that hard to explain to someone that you just swipe. It's like, how can I swipe? There's no bar. It's like I don't really think that that's the thing that people are like really fixed on It'd be nice to rid of that And I just wanted to know The Instagram app has been fixed on the iPad. They have actually done the thing they should have done in the first place. They've just made an iPad version of the Instagram app as it should work instead of that wild idea that they had where it was J Reels L can just how about just reels? No, theyve they've changed it now now just the home tab is the first tab So Instagram on the iPad is now just as it should be I saw a conspiracy theory about this. I don't remember who it was that I saw it. I'm sorry that I'm not quoting them. It may have been Parker Oolani. it was like, what if this is because it's this will be all the way work on the folding iPhone And I like that. I don't think it's accurate, but I like was bacy theory I saw that post. So I like a good I like a good theory like that of like, why on earth would you do it now? But I actually just think it's because it was a bad experience and they fixed a bad experience So this week was Global acccessibility Awareness Day, which has now become a tradition that Apple will show off some features coming to the twenty to the next operating system before WWDC. What could it be called? Exactly. That are the accessibility focus features because they then get their own time in the sun. And so people that care about these features get They see a lot of coverage about it. But then also we get and you know, like the media and us on podcasts, we spend time talking about these features where we otherwise may have just gloss over them more quickly if it was part of the WWC keynote because everything kind of get glossed over quite quickly after the keynote So there's a bunch of stuff going on here. And I think that the overall thing is Apple intelligence coming to a lot of areas of accessibility. I think that's one of the key things that is visible across a bunch of this stuff and I think that that's really great of like using models that can understand image and text and do things with those and people to be able to converse with these features a little bit more of allowing people who otherwise have issues with site or hearing or mobility to be able to navigate the world using their devices. I think it's really cool. So I want to run through some of the features that I found to be the most interesting. And there'll be links in the show notes to some coverage and Apple's press release as well if you want to read about everything But voiceover is getting some big enhancements. So voiceover is essentially the system that you can kind of listen to and kind of converse with the phonone in some ways. I'll just use the phone for an example, but it has always required developers to a little to do quite a bit of work and that can be quite spotty. And so both voice overver and voice control are being kind of enhanced to allow for the system to be more aware of what's going on and allowing you to kind of converse with it. But so they're using, for example, Apple Intelligence to better summarize images on the screen And you can ask questions of what you're seeing. as well, which is fantastic, right? So you's like, o, this is that and you can say, well what does this mean? And similarly, you can now use the camera with voiceover so you can hold your phone up And your phone can describe to you what's in front of you And you can also ask questions to your phone backwards and forwards about what it is seeing and it can use some context to understand that And like this is, you know In the conversations about potential airPods or glasses with cameras, these are the kinds of features that in theory could be enabled by devices that are permanently attached to someone inv visible. called Magnifier as well is also using Apple intntelligence to better describe what it's on the screen. So this is the app that can essentially use the camera or I believe it works on the screen as well to basically make everything bigger You can ask audibly for magnifier to zoom in on things or turn on the flashlight. so you can just have conversations with the devices that's doing this stuff And Apple kind of spent time talking about both magnifier and voiceover can be used as the action button. And there seems to be kind of like some additional features that the action button will be able to play in bringing up and activating these features I think the the one that I looked at, you know, there's always something. in the accessibility features that make people go, hang on a minute. And it can be either design or functionality And I think voice control this year is the one that is maybe raising the most eyebrows. So I'm going to read from Apple's press release The option to say what you see, is a feature they have, is great for navigating any app, including those with visual layouts such as Apple maps or files. Using intuitive language like tap the guide about Best restestaurants or tap the purple folder, this can help users overcome barriers when elements aren't properly labeled for accessibility Essentially, a user who maybe has some mobility issues that allow them to control their phone correctly can can just speak to the phone askk it to select stuff Now that is very interesting because That is agentic operation of IOS, essentially Isn't Isn't this is the paper they published like a year ago though. we talked about Ferret too Yes. Wow. I would not have remembered that as a good pull I think this is very interesting, right? And like they show a video in the press release of somebody like asking like Select the orange folder there's like a green, orange and yellow folder and then you see the phone select the orange folder. And so I think This kind of stuff is really interesting because it is showing The iPhone and the user being able to have a dialogue and do things on the device, which is kind of at the moment, I mean we're going to talk about Google IOen in a little bit, seems to be one of the big new areas of tax R of like controlling devices and how that's done. So this is very interesting And then a couple of other things that were I thought were called accessibility Reader can now handle more complex page layouts that may include tables, columns, and images so it can this is essentially giving people more ways and tools to be able to change fonts, change sizes, you know, maybe Like for some like dyslexic fonts as well, like the fonts for dyslexia. they have those in there, so it can kind of adjust text. It can also do translation You can generate subtitles for personal videos prettyretty cool. And then one that that was just like wild and they have a really good video about this in the press release is Apple Vision Pro wheelchair contontrol where essentially For wheelchairs that can connect and Apple's working with a couple of partners about building a connection between the two things You can use the eye tracking functionality of Division Pro control and move a connected wheelchair. So it shows the UI that the user is seeing where they're essentially got like likeike a D pad, I'll say, you know, like like video game controls. and using the eye tracking, you can like look at the go forward and it will go forward, look at left willurn left The video is astounding. I like to see this in action And it you know, really made me think like the vision Pro is not the device for this, right? Like because it's huge. And you don't want to be wearing it all day every day. But you can imagine a world in the next few years where this functionality is built into something more wearable like all day it's going to absolutely change the lives of people because they'll be able to have autonomy in their movement where they otherwise may not have had like , I Every year this is interesting stuff, but I think the combination of what they're doing with Apple Intelligence and then this Vision Pro thing, I think is like I think it's super cool. Yeah I mean, Apple does such a thorough job with all this technology and One thing that, you know, we've known from hearing interviews from people who work on this at Apple and our friends who cover it more closely Apple doesn't develop these features in a vacuum. L they're working people whose lives B. changed because of this technology. and It's very grounded work in a way that I really appreciate. And yeah, the Vision Pro thing definitely jumped out at me is like, that seems a little a little strange, but yes, so much of this is like building towards the future Right And ye. I think it was on I think it was an upgrade. Y'all talking about the airPods and mentioned the door detection cknled that Apple announced two cycles ago with this stuff All the stuff builds together. Th aren't necessarily discrete features or technologies they're building these things together in a way that they can be used together in meaningful ways. and like I always find these posts so exciting and encouraging because Apple really does care about this stuff And it I think it shows in the way they go about developing them This episode of Connected is brought to you by Squarespace, the all in one website platform designed to help you stand out and succeed online whether you're starting something new We were scaling something up Squarespace gives you everything you need to claim a domain name Showcase your offerings with a professional website Gr your brand and get paid all in one place One of the things I love about Squarespace is that it can grow with you over time. Say that you've got a site that you built on Squarespace maybe a couple of years ago and Now you've got new branding for your company or your project Well, you don't have to go around page by page and Change each color and each font and make sure it all lines up Be Squarespace desesign tools are really easy to use. You can make site wide changes really easily and really quickly. It's easy to understand what's going to happen. their design tools But maybe you're not designer Well, Squarespace's AI enhanced website builduilder lets you quickly and easily build a site bespoke to your business J input some basic info And it's going to combine that information with Squarespace's professionally designed templates However you start, you have beautiful design options, no experience required And you can manage your videos easily. Maybe you've got how to videos. Maybe you have a video course and you want it to be available for sale not to the general public. Squarespace can handle all that as well. Vide libraries are really easy to build out Paywall content is easy to manage. perfect for online courses, tutorials or premium workshops. So head on over to squarespace d. com slash connected for a free trial And when you're ready to launch, use the offer code connected ten percent off your first purchase of a website or domain name that squarespace d. com slash connected and the code connected for ten percent off your first purchase Our thanks to Squarespace for the support of the show All of relay It was Google IO Heo I'm gonna be honest with you guys ot watch the entire Google IO keyote. Meither. you'd be dead because I find the whole Google Io keynote to be to be rough What I love and wait for is the Vverge. The Verge posted a thirty five minute video, which was all of the big announcements. I watched that And I think I got everything that I needed from that and then reading some articles. but The whole Google IO keynoto. I mean, it suffers from the problem and I actually think is evident in what we're going to talk about. of too many teams working on similar things Y who all get their time on stage. which is just like a fundamental issue of Google and how it's and how it operates as an organization. I think Ben Thompson referenced it as like the the keynote is the structure I forget he has a very funny turn of phrase basically like You are showing the internal structure of the company. through who's on stage and I always always been that way or at least in the last, I don't know. fiveive or ten years and inccreasingly feels like We are not showing products We are showing like projects What's in the company? And I think Google has struggled in the past. I think Definitely this year or two of like, We have this technology, we have this idea of a thing But we don't know how to like get it into the world. So we're just gonna talk about it and like maybe it'll show up or maybe it won't it won't. I mean, sort of famously In the past, a lot of IO announcements are just vaporware. They just never sh or they ship to a very small number of users as an experimental beta And then it It goes away because it doesn't What lics is like Google Photos removing a chain link fs. Chain link. Where is it? It' a demo they showed multiple years and I'm not sure it ever shipped. Yeah. they just keep showing like, Oh, look how great this is, okay Well like the the callal holding thing that they did, you know, like all that kind of stuff. Now, I will say like There some stuff there's some stuff from Google IO that I found really interesting and like obviously a lot of it's Gemin stuff And we'll talk about those But my overall feeling watching this thirty five minute video was a feeling just of great despair at some of the stuff that Google is doing. The majority of my despair is in their video and image creation tools and and the applauses that these features are getting from the crowd and like and I don't know who it is That is applauding. I guess I guess it's people that work at Google because I just I really struggle to be excited. about image and video generation. do and they have like, multiple systems, right that are doing this. So we have Gemini Omni, which is a new model that you can use in Gemini. Oh in Google Workspace or Google Flow Three distinct tools that do the same thing. Welcome to Google. And it's like, what Pix is new? It's like, why are we doing a new one? Why do we need a new one But like the thing for me, like they showed some demos of what Gemini Omni could do and teechnically seems impressive. Some of the demos looked better than others. they were all presented as looking amazing, but they didn't. Oh look even, you know, some of them are very convincing, some of them really weren't The question I have to ask is why Why do we need video and image creation I don't I feel like I don't No Why it should be so easy to create fake videos. and even now like they're really like, ah Look at this video I made and I'll just change it. and look how great this is. It's like but I don't know. I' extremely problematic. Yeah. like I don't know what what this does. L A lot of the other stuff that they're showing you can see how and I understand and not everybody likes it and I get it. But you can see how people's lives could be made more productive potentially by some of these tools I don't know what this is doing is worthwhile any worthwhile application weighing the vast problems willill be created. Bye This technology existing they follow up, you know, they show like Ohh look at Chairan Nomi amazing. and like, o and by the way, like We're also now building tools that will help people understand if the content that they're seeing is generated by AI It's like, is that supposed to like give you an out that like Oh people can just ask Google if this video or photo is real or not and either you'll have the credential check stuff that people have to opt into, but I'm still not sure how that's actually supposed to work, but whatever. But then they're like, Ohh, you know, we're going to use a bunch of functionality to be able to detect it. But like that's all great, but it still requires people to think Ask issue is the people that think Ask are not the people that you need to worry about, right Like if somebody sees a video or an image and they're like, I'm not sure if that's real, awesome You're good But it's the rest of the people who are like, well, that's real And they're never going to ask Google if it was Yeah real or on. Yeah. media literacy continues to circle the drain. Yeah Because you literally cannot believe what you're seeing anymore which is very difficult. They get because at the scale that it is created People could always make fake things But it was hard to do. Yeah, like I learned Photoshop in college. I can make some fake stuff in college and Photoshop, but It was not at the scale that it is now and to go back to something we talked about months ago when Grock was just undressing people It wasn't built into the social networks because there weren't, you know, like It's so prevalent and so easy to create and to see and to find Um, it It is a different thing than oh my God, you know, Photoshop. like they're different the levels are so different. And if you can't see that, like I don't know what to tell you I would expand your angsty question though to more than just image and video generation, although I o holy have it about those things At some point we're going to talk about some of the frankly bonkers things. Google Deep Mind CEO said If you really All of these companies all of them. Many of these companies will say like This is very powerful technology and it com basically, it's the it's the sppider Man quote of like with great power comes, great responsibility But I don't believe any of these companies will be able to exercise the responsibility that they say is needed. If you really think I'm just going to read it because it's so stupid The u If you really believe that we are standanding h in the foothills of the singularity Yes. If you really believe that Jim, you're going to have to Bleep this Stop Like just stop But also, what does that mean Demis Sasabis. what are you at? Like? Did anybody check with him what he was going to say It Sounds like a pretty dope name for an album. It does. It sounds like an album Muse would make? No no, it reminds me the muse Of the Oasis standing on the The Oasis did standing on the shoulders of giants. And this is like We have OSis at home and it's Demi Sasabis saying we're standing on the foothills of Singularity. Because it's like what what does that mean Like what does that actually mean? What What is the singularity in this scenario? Because he says that after saying a bunch of really interesting stuff. You know this is the kind this is the kind of quote that the previous sery guys swiming toai would say. Yes. Be it's like he says that after talking about like, oh, we're going cure every disease and it's like, Yeah, you know what If you can do that, like awesome, you know? like I'm on board. If you can cure every disease, sure I'm in this And to Googles Oh, the alphabetets' credit, like and u uh, this Demis do you I don't know how to Dis Demis Hasabis Hasabas. Yeah is involved with I of isaphoric labs, which is like trying to use this technology to quote, I wrote it down Our mission is to reimagine the drug pro the drug discovery process. and the goal of one day solving all diseases Um, That's fantastic. and like I'm on board. Me too, right Demiis Sovis is not an idiot. No no he' a noobel prizing chemistry. Yeah. Yes. Like he's smart You're saying He's smarter than the three of us put together Yes. Oh, for sure. But that doesn't mean you don't say unhinged things on stage. Yes. But my point is ' trying toin this back in. Um If you really think If you really think AGI is on the horizon, if you really think, if you're anthropic and you really think that mythos is too powerful put into the world because it's going to break all software humanity has ever written. Yet you keep going. They all keep going Right? It's It's like someone like you go to somebody and you're like, hey, man, like You have this really destructive habit in your life It' be we would love you to stop, you know, whatever it is smoking drugs, using windows, whatever it is And so many times those conversations heartbreakingly end and somebody saying, I know it's bad for me, but I'm not going to stop And I just The whole time I watch is I watch the whole Vverge thing and a bunch of other stuff And that was the feeling I had the whole time It's like if you really think If you really think we're this close or that it's on the horizon or on the foothills or whatever analogy you want to use If you really think, oh, we need to build a whole system So people understand that the video they're watching is real or fake. Stop making the video creation tools. Yes Now what they what their answer is to this. as well. No one else is going to stop But we are the ones in our eyes who are going to be responsible with it. So we have this was why openpenA was founded and why, you know, they were initially nonprofit. It was we're going to build up this technology so we know how to control it and we know how to be put responsible saafeguards around it. So when other people inevably get here and do similar things, thenen we have an understanding of how to manage it But that's not how this goes. It's not how it's panning out at all. No, No. Yeah, they're like they're like the meme of We're all looking for the guy who did this and they're the guy. Like ye, they created a solution to a problem that they created in the first place. So I just looked up, by the way, the definition of the singularity. please. And this is being provided to me by the AI overview, which is just like What are we doing A hypothetical future tipping point where artificial intelligence and technological growth accelerate so rapidly that humanity is irreversibly transformed Now to try and throw C service a rope here. Sure I think what maybe what he was trying to say. is If we can get to the point W we cure all disease That is essentially singularity moment because we are transformed. Humanity is irreversibly transformed for the positive at that moment. that like we don't get sick anymore, right? which is like Again, there are many other issues that I think would be caused in that scenario, but like Well it's good, right? It's in theory is a good thing. right We need to manage that as a human race But like that is a good thing. And if you just follow that thought through, it's like, yes, okay, if you believe that's true, we are standing in the foothills of the singularity. However, I recommend you do not use that language because it sounds unhinged. And also the singularity sounds Really scary as a phrase. And it's like, are we in a simulation kind of stuff Yeah, I'd I would love to know if anyone knew he was going to say that at Google before he said it. Like was anyone checking his slides? Like man, what thing to say? So let me play Devil's addvocate for a minute. Please. pitched Gemini Omni as a world model which I think is a reallyally fascinating technology. So a word model is not just a lar language model, that's obviously where it all starts It's not an image model. It's not a video model per se, although it can generate videos and I'll get to these in a minute. It's can build a real time simulation based on real world physics. That's like the short description of it So if you're thinking, we want to achieve the singularity and we want to be able to cure every disease I think it's obvious that you start from a place of well, let's teach AI, how physics works So that in a not so distant future, when we want to put in a nanoc camera inside of your body and the nanocamera needs to analyze blood flow and muscle structure and bone density, it starts from a place of understanding how real world physics works My problem with all of this is that I think the if they're not lying about this. so Big emphasis on this sentence. If they're not lying about this, that is a noble goal. it and it makes sense to start from a place of giving the AI knowledge about the world, about physics, about how walking on solid ground works, how gravity works, how the sky looks. like it all makes sense that you start there My take is that Two takes First Not every bit of research needs to become a product. Yes And two All these AI companies have a serious PR problem Oh yeah. They are petreine products that Journalists and employees in the audience because they are technically impressive. Normal people in the real world, they look at that stuff and they're like, look at this Jam, I'm sorry. canan you also blip this?. Real peopleeople in the real world are looking at these announcements and they're thinking look at this Like look at what are they even A am I going get mine in today? And have And you have Dario Amode saying ah, pay attention. The AI is going to take all of your jobs in the next twelve months. First of all Somebody needs to make Dario stop saying those becausecause then it's also like, and we're going to give fifteen billion dollars to Spaceks. It's like, okay, are you doinginking the jobs? Like who' taking the jobs? Not every bit of research needs to become a product. like you can build a world model without also saying ah, and we're making Gemini omni based on that research and now you can make ph realistic videos that look just like real life. No, just don't release it. Just don't make a video product U so I think there is an inkling of nobility. to all of this If you can make the AI autonomously research and create Cures for diseases, That's awesome. To go from that to just type in a prompt, Hey, make me a fake video of Osama bin Laden actually being alive and chilling in Paris, like no, no. Beuse it's like even the demo they show, right? Like it's interesting. Here is a marble moving. and it understands Gravity understands friction, understands momentum So we can make this video now Nobody needs it Like, but the things that Gem andi Omni can do in other areas Oh, that's really interesting. But like why I agree if you' like you don't need to apply every possible outcome to this technology I could somewhat understand why these companies were pushing so hard on photos and videos when they didn't have an interesting product Thank a couple of years ago or whatever, you know, can this all came into our lives because of Dali, right? Like that was the first thing that And what was the other one M mid journey? Yeah. Yeah, But Darly was first, right? Like it it broke onto the scene before Chat PT did Like it was a thing that like people could see and use and we were understanding it was being generated, right? And like that technology was like Okay, like here's the thing you can do. But at this point other things that this technology can do. That is more interesting Why do we continue to advance the video stuff And like there was like a phrase that one guy said' like, this is the Nano banana moment for video. And I was like Shut up. Like just shut up. you're talking ridiculous nonsense the Nano banana moment was like, okay But it's not made our lives better. You know, the fact that Google made a model that can make images look really good. Also how have we reached the point where that's an actual English sentence That's allowed. also like I am British so I can say this. There was something this guy was British too and there was something about his British accent saying the Nano banana moment for video. I was like, no, that's nano banana. You can't say this. America can say this. You can't say this. You have been living in America too much Banana banana Nan Nano taco banana, sayay that, sayay that And now now talk about. All right Less Push this aside Yeah, ye existential huge broom off the table. Push this aside because there were some other things when it's One of the things they're showing in a A bunch of their demos is Gemini of Voice. Lots of people were dictating to their phones in Jem and I was doing things like long rambling lists of tasks, corrections and like they were purposefully flubbing stuff and you know, like just It was really interesting where sometimes like a person in one thing was like they're dictating an email And then they start giving instructions to Gemini to dictate and give more instructions And the model could understand all of that, right? It knew the parts that needed to be presented as text and an email. It knew the parts that was go and grab these images from Google Drive, like or whatever. like and it did all And This kind of thing is the most exciting to me right now of a future where we can have a dialogue with our computers and they understand what we want them to do And I find that to be quite exciting that like in a few years time I could communicate with my Mac. like ask it to do things and it would do that. Like, I find kind of technology to be really interesting And then this kind of goes into a product that they're because they got to keep everything needs a name Gemini Spark. This is their open claw And so the way they set it up, it runs on dedicated virtual machines. It's running in the background. You can use it in the German I app currently integrates of their tools. it will have MCP integrations later on so it can integrate third parties The fact that now everyone's kind of doing this Yeah This is more similar to this is actually more their take on perplexity personal computer more than anything. But that was a take on they're all trying to do an open claw thing, but in a way that's not as risky, right? Yeah, yeah. The idea that you have this agent, you connect the agent to a bunch of tools and the agent runs in a sandbox can do things for you and sort of the industry is kind of settling on this terminology where you assign tasks to an agent, whereas you have chats with a chat bots So in spark and in perplexity, you create a task, whereas if you go to Gemini or ChT regular, you start chats. Um I'm very skeptical Of Spark, mostly because the Gemini models historically have sucked at tool calling and integration with external tools. We spoke about that in terms of Apple intelligence, right? Exactly. That could be a weak point And this Gemini three point five flashed doesn't look that exciting in benchmarks Um I mean, it's fast for sure. It's more expensive than before And it continues not to be state of the art for calling external tools. U So I'm I don't think it's gonna be that Great. They're launching with a subset of integrations and later on, they will let you plug in any MCP So I don't know. All of these companies, I mean Google, SpaceX slash XAI, whatever it's called now. They're all kind of late to the agent party, so to speak, and they're trying to catch up now. meananwhile, open AI and anthropic are so far ahead, It's kind of ridiculous That makes me, you know, it's like, you know, we're on the precipice of WWDC again, right Mhm and It's like, oh, are they Doid they have anything like this? Now There was, I believe a rumor a while ago about the idea of talking to your mat one of the Mark Garman rumors and it being able to do stuff for you. But I just I feel like, you know We're going to do WWTC and they're going to have a bunch of stuff But then the thing everybody else is doing, they don't have an answer for it. And I figure we're just this is just what it's going to be for years just because of the way that they developed apple. Like I just If this is the thing Do they have an answer to this thing and if we go through WWBC and then all the questions are, where is Apple's Aentic serory, right Are they going to have to change how they work They want to be in this world because clearly the way Apple works If that if this scenario plays out, the way Apple works just does not fit with the way the AI stuff is developed And I just wonder if they're going to have to adapt. basasically releas stuff more frequently than once a year We'll say Hm. Google search looks like it's going to become rough So There's a bunch of things going on with Google search. And I think the best way that I can describe it is they are just putting more and more AI into search So like the AI overviews are getting bigger and they're going to have more stuff in them They're going to generate UI for you. They're going to generate code for you and put it in the Google search so you could like have these visualizations They're going to have agents that can keep searching for something in the background for you. It feels like it is going to become harder and harder just find The links. As a website owner, that's my reaction Yeah That girl's not great It's wild to me. I mean, so much of this stuff is bananas, which we've we've spoken about, but Changing searches Wild. That's what pays for all this stuff and You know, they've been, especially on mobile. The Google AI overview, like it will take over like modally when you search just in Safari. like I'm not using the Google app Maybe it's different there. But making this the primary interaction for search. Wh I understand it from the business perspective of like, wow, chat GBT and others have like made it a pretty good case that LMs are useful in search and research and those sorts of things. I mean, that's the thing Google does. It's those it's those links and the ads that come along with them And fascinating that they're willing to change it. I think it's the, I mean It's like fundamental to who they are as a business Wild I mean, you've got to assume that the plan is they'll just start putting ads in the AI overviews and that's how they pay for it. right? Yeah The AI overviews are making the ads less visible, which means they will just have to find a way to put the ads in the AI or just put the ads above the AI over or like whatever. because I agree with you. They're doing this now at a certain point the rubs is going to hit the road, right? And they're going to have to find ways to monetize whatever Google search is going to be Um seeeeing this stuff was like maybe gave me the first time where I was like, maybe I don't want to use Google for Google anymore. Yeah at the same thing. And I think I'm not like I'm not like this is not super close to me, but I feel like And I've explained this before. If I Googling for me usually is I have an idea of where I want to go. And I use ChadGPT for searching for things that I don't know about yet and And so likeoo when I'm Googling, I usually have a destination in mind That is very like in line with showh me ls links of stuff And if they're going to make that harder and harder, maybe I just need to find another search engine where I can just find links to stuff, you know because that's what I'm looking for out of that product because I don't find Google's AI overviews to be suufficient for the bigger stuff because they try to serve it too quickly. And I think that's the fundamental issue with the AI overview They try to serve it immediately and that means the quality usually isn't so good. That's my theory anyway, of like why AIO overviews much worse and say if you were searching for something in Gemini So yeah, I don't know what they're doing over there with Google search. It seems like a lot and just in general Google St win Google's doing some stuff. Yeah That's That sums it up. They're doing some stuff This episode of Connected is brought to you by Steam Clock If you're like me, you have opinions about the quality of the apps you use You know the difference between an app that feels right and one that feels off Steam Clock software builds mobile apps for companies that care about taste They're a design and development studio based in Vancouver, Canada And they've been shipping iOS and Android apps for over fifteen years theirir clients are growing tech companies that care about mobile donon't have the in house team to build something great SteamClock works with companies to level up their app So we can go from it's holding us back to it's really pulling its weight Some of their clients discovered the hard way that vibe coding your way to the app store is not a product strategy SteamClock has deep experience shipping apps for IOS and Android. They get these platforms and they're good at helping companies figure out the right technical approach for their situation Apps built by SteamClock have been downloaded over ten million times And they've helped five of their clients through acquisitions So if you're building something and need a mobile team that cares as much as you do Steam clock is where to start Go to steamclock d. com slash connected to get in touch That steamcock dot com slash connected. Our thanks to Steam Clock for their support of the show All ofay. Federica, you have huge news. You've been working on Sortcuts playlground for a long time. It's been fun to follow that Behind the scenes. Um So tell us, tell us about this project Are we standing in the foothills of the shortcut right now Yes. Now you are the on the top of the hill, actually. Oh wow. Yeah. Yeah Look around you, one day, Micha all of this could be yours. Oh yes. Thank you Cryal lightight touches Yeah So okay, so la I launched a bunch of things on MaxSys today. The highlight is the Sortcuts playayground plugging for Cot code or open AI codex That is free on open source. You can go to the Gitub repo and you can download it. You can take a look at the code yourself. You can just ask Claud or Codex to set it up for itself and it'll do it And this is this is a pluggin That lets you create on your Mac, obviously, because these things are Mac only. It let's you create real Apple shortcuts that you can open in the Shortcuts app and add them to your library. That's the short version. You just type a sentence, type whatever you want. likeike, hey, I want a shortcut that gets me directions to my next calendar event or I want a shortcut that shows me how much caffeine I've had today You type anything, you get a shortcut. Obviously, there's a bit of latency involved because you're having an agent create a shortcut for you. depending on the model that you use, the complexity of your idea You're either going to wait a minute or if it's really complex, you can wait, I don't know, ten, fifteen minutes, depends on what you want to make. At the end, you get a dot shortcut file on your desktop You double click it, you add it to the shocutsap and that's This was a lot of work.. It like how does this even work? Yeah. yeah. so. is like It doesn't feel like it should you know? Yeah, shouldn't work. So this all started in January when I came across this skill, not a plugin, skill for cloud code. Basically just a bunch of markdown files from this person Drew on GitHub and the skill I was fascinated by the idea because they It was basically a bunch of markdown files that Hot clot code how to put together shortcuts. And this is like the core of the whole thing cuts behind the scenes I've always been Fancy XML files. Aren't we all really? Aren't we all really, if you think about it? Technically, an RSS pcast our podcast is an NL. Exactly It's always been XML And realistically it' always continue to be So this has always been the case since the workflow days And even after Worflow was acquired by Apple, a shortcut is essentially an XML file gets signed into a proprietary key list format. itself gets saved as a dot shortcut file Right now You can export any shortcut from your shortcuts app as an XML file skill that I discovered at the beginning of twenty twenty six essentially contained The information for Claude to learn how the XML syntax worked behind the scenes, for different actions variables, parameters conditional blocks, repeat blocks, everything And Cock code could say, well, okay, I know the syntax, Now I'm going to put together a new XML file for a new shortcut I thought it was fascinating And I started testing it And It was very broken That skill It wasn't working half of the time. It was hallucinating actions It was creating actions without variables. The shortcuts didn't work. But there was something about it that I thought was worth exploring So I took that foundation and started working on my own version of the skcale. And I realized, well, okay, Hold on a second. If this is based on XML knowledge, And if I can export my entire Shortcuts library as XML Why don't I just teach code myself with my own shortcuts So starting from a modern foundation of my own shortcuts from my own app Maybe it'll find a way to update its knowledge. from that skill And it did First, I gave it my entire shortcuts library and it updated the skcaill based on that And then so he started snowballing from there. The next step was I remembered that the entire Sortcuts database If you have a Mac is actually freely available for you to copy, back up and inspect. It's an EcUite database that's on your Mac. And you can just copy that give it to an agent and say, Hey, take a look at this And I did that So the agent took a look at that and understood actually how variables worked, how app intents were named how parameters worked, all of those things And then I said, well, okay, so I took a look at my library, took a look at my Sortcuts database on myQS What if I give it the entire Mac Stories Shortcuts archive And see what happens And so from there sccraped my own entire archive on Max stories Downloaded three hundred shortcuts Repeated the same process of overver And in all of this, so while all of this is happening, I'm creating shortcuts, right with the scale It wasn't a plugin, it was still an improved skill. I'm creating shortcuts and it's working. It's starting to make shortcuts And I'm giving it feedback So for Three months or so every day I would sit down Type in some ideas, create shortcuts, install those shortcuts Go back test them. This is all manually done by me Test them Past the feedback back to that point I moved from Couot code to codeax. pass the feedback to Codex and say, hey, look at this action, it was not configured properly is other action, you actually hallucinated a variable overver and over and over. U So these for about three months. then you did some model training Y Yes. Yes. I actually did a Ka lightweight version of reinforcement learning, basically Um Then I had another idea where once I reached the point where the skill was like Really good It was a really good skill But sometimes it would still break something. It would still hallucinate an action, a variable I remembered that one of the popular techniques in the AI programming community was these so called Ralph Luke I think we spoke about this unconnected before. Uh, Ralph is this technique now also a plugin that you can install where you basically put an agent in a loop where it validates its own work And if the validator fails, it gives the feedback to the agent and says, Ah, you wrote some code, but the code is failing here. Fix it. Over and over It' called Ralph after the the the from the Simpsons, Ralph that Ralph Wiam Yes, makes mistakes over and over and persists no matter what Um, And So I thought, well, wouldn't it be funny if I sort of adapted the Ralph Foop technique? for making shortcuts. Uh so in the in the In the plugin, you will find my take on the Ralph loop. It's called a craig loop U And it was not named alsoough if a popular character from from The apple. The crag loop is this huge Python script that anyt time the agent tries to say, okay, I think I got it. I wrote a shortcut draft to disc The validator kicks in automatically and tries to validate the shortcut If there are errors, it tells the agent Yeah, you wrote a file, but there are such and such errors. G back And this this is the loop And it's why Shortk' playlaygand can be so accurate for a Amateur reverse engineered project because there's the crag loop behind the scenes that over and over any time Coud or Codex, say I think I got it. I wrote something to this the validator comes in and like h, let me take a look at this. Does he validate? No, hereere's the feedback. go back Until the validator says yes, it passed. point Uh pllugin uses the actual Apple made native shortcuts CLI signs a shortcut And that's why you're able to install it in the Shortcuts app Um The fine so that was My big breakthrough was that idea of putting a validator pututting the agent in a loop Mhm The second breakthrough was using codex In a loop for multiple days Puter use to create a thousand shortcuts hing Codex Kodex thought about Kodeax took a look at the entire documentation. There's a lot of documentation in the skcale Take a look at the entire skcale made a plan to create a thousand shortcuts for all the possible categories and all the possible actions And it worked in a loop. to create a thousand shortcuts And then it used computer use to install those shortcuts, actually double click them openen them in the shortcuts app Take a look at them Test them And if something was visually broken or functionally like if something wasn't working It would see it on screen and repeat that process over and over until that particular shortcut, out of a thousand shortcuts, was working So it worked for multiple days on this on its own. and every once in a while I would screenshare, loggin and see what was going on just scimine U And that's and that was the other breakthrough, like actually putting the agent in a long running loop for multiple hours at a time. And u Then about a month ago I thought I had it So I started putting together everything that you see on MacSource today, the redesigned shortcuts archive the dedicated mini website for Sark' Pray ground And there's a Companion shortcut, that's for paying members only U That's also like the crazy part. The companion shortcut it is called Shortcuts playlayground rememote because I wanted to convey the idea that it's an optional like remote experience thing. That is a shortcut that creates shortcuts on device. You actually run it on your iPhone and iPad and you type in an idea And a few minutes later You see a new shortcut import window on your iPhone or on your iPad And you may wonder how that's possible actually talking behind the scenes to your plugin installed on your Mac via SSH. And it's getting the shortcut file As drum roll Base sixty four. Oh. No wow. Yes And that's why it can transfer that shortcut from your Mac to your iPhone or iPad. That's a companion shortcut. It was very difficult to put together.. There's an automation accademy guide and sort of story how I did it it's veryy complex, but since I started. fromrom an open source skill And since I Throughout this process, I drew from a lot of previews open source projects for shortcuts. U That's why I decided that the plugin be free and open source for everybody to use Um There are a lot of nice things about the plugin. like for example you can ask for shortcuts to be installed and to have a specific color to have a specific icon glyph And there's actually support for icon synonyms. So it actually supports the official shortcuts Glyph library. but If you don't want to be using the exact name of an icon that Apple is using, you can ask for a synonym of that icon U the shortcut knows how to make health kit. U Sorry, the plugin knows how to make health kit actions. It knows how to make complex actions with conditional blogs, API calls Anything you can imagine you can ask. And the final thing I will say is Since It's running in an agent on desktop This thing is running on MacS, so it can use a terminal. One of the nice things that I've discovered is If you ask Shortcuts playground who's like, can you make me a to doist the shortcut for I don't know, getting my tasks for touist today and let me choose which ones I want to reschedule to another day But The agent will actually test U The to do is API on its own for building the shortcut This was actually one of the best practices that I sort of infused myself into the shark the pluggin by telling it Anytime the user you asks you to make a shortcut that involves a web API Try to test the web API yourself so that you don't hallucinate URLs or API responses So I don't know. I'm pretty proud of the work that I've done and likeike I wrote in the conclusion, I fully expect to be Sherlocked in you weeks by Apple Well mayay, right? Mbeay Mark Germman said In the updated Shortcuts app users a presentage of a prompt asking what do you want your shortcut to do along of a text field to describe the request, the system then automatically builds and installs the shortcut on device. Sounds pretty good, but you never know, right? likeike with this stuff. you never know how good it's going to be or not. So Maybe there'll still be a case for short class playground even in a twenty seven release. I this This is something that is like definitely very impressive and also hard to get my head around, right? because it's very complex and maybe your craziest project yet, honestly, of which you've done many I've done many, but I think what's different about this one is that It's actually useful for other people whereas in the past when I did things like the MacPad, for example, it was more I don't wantan to say self referential, but it was always to prove a point myself and share a story about myself Whereas I think what what I'm what I'm trying to do now and what I will continue doing is release more like free and open source things for other people to use Um, becausecause I feel like I want to be able to People ask me the question of like, yeah, sure, but What have you built with AI that's actually useful or that's actually making you money I want to be able to point at stuff And and I'm very happy today because when people Now when some people ask,h, yeah, d, but what have you built with AI? is actually useful, I can point them to something ical the new shortcuts are arve are Max stories The plugin the shortcut for members. like that is real stuff that is helping our business. And Of course drop of it all I learned a lot I learned things I had no idea about. and U It was the first time this was the the the real first test for this new way of working that we have a max sties whereet there's me or John. And we have our agents And At the end of the process, we have our developer That's who's reviewing all the work of these agents and who' actually taking the output of our ideas pututting them live on Max stories So like It' this new way of like Hybrid work. I don't know how to describe it If I like the truth, like if I didn't have agents There will be no shortcuts archive, there would be no dedicated website, there will be no plugin I don't know. it's all very different for me People seem to like it. And uh, And I don't know, maybe Apple won't I do wonder. I mean, this obviouslyve know you've been on this for a while and there's definitely been some conversations about like Is this thing going last? Like Will this see out the week? We'll find out, you know, one way or another But it for as long as it exists, it is Very, very impressive and complex. Thank you. Um and is a This is a true Federic ofver teachi project This one Uh Yeah, my next one, so I'm trying to release a bunch of these things before WWDC. And there's a I have Two more projects that are ready to go. I think realistically I'm gonna have time for one. So my next one is I mean I can just mention it on the show. I built a command line interface for reminders on the mapm And it's the the I mean, not to brag. it's the best reminders CLI Um, on available I mean, once it will be available because it's the only one that supports all. and I mean And of the reminders features, including the private API ones. A hand yes. You can't say that out loud. Well private. It's private It'll be free and open source and That's what the Mac lets you do Yeah, man. when you have when you have stuff available in Finder for you to inspect. And tinker with That's what happens you promise you're not going to bring the singularity Are we standing in the foothills of the Federico singularity? You can stand in my foothilds. That's all I've ever wanted Okay All right. Well, I think that about does it for this episode of Cnected. If you want links to the stuff we spoke about There are and they're in your podcast players like Where my voice is coming out of, there's links in there Go check them out. They're also on the web at relay d. aM slash connected slash sixz four A couple of other links you to know about. you can get connectedfeedback. com headad over there. you can leave us a note. You can leave us a nice note. you can leave us an anonymous note You can tell us your WBC predictions, just have Ea in there. orr tell us what's going to happen at WWEC. If you know that, I Yes now takeake it You can join and get longer ad free versions of the show. We do those each and every week. There's extra continent at the beginning of the show, no ads, and then we put titles at the end You can get that for just seven bucks a month at get connectedpro. co and again, that link is in the show notes If you want to find more of us, We're online, Prince Flexi Federico is at maxstories. net. We just talked about his new project. And the promise of many more to come check's calendar in three weeks. so Make Macstories your home page, I guess,' what I' saying You can find a mic across a bunch of other shows here on relay and his work at Cortex Brand and he blogs at the enthusiast d. net d I've been your attorney General Flexi. And you can find my writing at five twelvepixels. net I'd like to thank our sponsors this week for sponsoring the show, making it possible Squarespace and Steam Clock And until next time guys, say goodbye I do that you Oh no at the end too. I gotta get it in. Come on, we're all giveiving one. Yeah, okay.