Modernizing Xcode and App Intent Domains

Hello and welcome to another episode of App Stories. Today's episode is brought to you by Things from Cultured Code. I'm John Vores and Federico Vitichi is with me, Hey Tichi. Hello, John. how are you? I'm having a great day. It's a holiday here in America and it's raining. I think you know I don't I feel like the worldre working on a holiday. Oh it's fine. I work on most holidays Unfortunately, but that's a whole that's a story for a different day We are going to continue our wish list episodes. and today is I made it very broad. I made it automation wishes because This is both going to be about shortcuts, but it's also going to be about Apple intelligence and you know those things are all part of the same ball of wax really, but But you know, there's a lot of aspects of this that I think go beyond even just Apple's technologies that we want to see implemented as part of some of these OS updates that allow party tools to come into the OSs as well. So We're going to kind of talk about Oh Do sound good. Yeah, yeah. Where do you want to start with this Well, want to I guess I wanted to start a little bit about sort of what we expect. I mean, most of this is going to be wishes, things that may or may not actually happen, but things that we want to have happen. But I do think there's some expectations given all the rumors swirling around about Google Gemini being the basis for some of the Apple intntelligence and other things. that I think we can talk a little bit about our expectations and what and what our wishes are for know the expectations that happen to be out there right now. One of those to me is I think we will see foundation two point zero, which is Apple's foundation model that runs on device. I assume that that will get And one of the things I'm going to be looking out for, I think, too, is some kind of indication from Apple about whether they're going to be more public about the updates to this. I mean, they have updated foundoundation in the past before kind of behind the scenes a little bit, I believe, but they haven't really about it that much and I think with two point it would be, you know, given the way people are sort of building in public with various AI models. it'd be nice to see Apple tout what's happening with the changes in the model over time. So people have a sense for whether it's proven or not Yeah, I mean, for sure we're going to see the mayaybe the second generation of Apple foundoundation models And That would obviously be the C and and the So the feature that Apple would try to sell in the twenty sevenh family of OSis And so I've been thinking about how they are going to roll this out. I think there's a There's a scenario that I sort of foresee happening at WWC, which would be version two point zero of the on device Apple foundation model that is an Apple trained notot Google trained but an in house on device foundation model 's going to be smarter than the first one likely I would say likely a bigger model, maybe with a bigger context window, I think right now And they have a four thousand token contact window. Very small, ye. I think at the very least I would expect eight thousand so double the contacts window for the on de vice model. Obviously an on the de vice model needs to be small and needs to be high performance and low latency. So do not expect huge model running on device, but a second generation of that model I think could be really nice for on device cllassification tasks and coordination, some lightweight orchestration uh, even So I think we're gonna see G two of that. And for the private cloud compute, I don't think we're going to see an Apple made Apple intntelligence foundation model on private cloud computes, that's going to be the Gemini backed version. right? So I would expect a Gemini based Cloud foundation model that is based on Gemini fine tuned by Apple. Label as Apple intntelligence in the cloud or something like that with the same privacy guarantee as the first generation of Apple intntelligence in private Coud compompute. I think that's going be a big chunky model, one million contacts window. Likely what is Gemini a one point two trillion model? I think so. Yeah. something around there. I expected personally, I expect it to be based on Gemini three point one Flashmperro If I were to guess, I think it's going to be based on Gemini three point one flash, not three point five flash U If only because work on, I mean, Germinite three point five flash is not even out, I believe at the moment. Right. It's very new Yeah, I would expect that Apple has been working with three point five three point one flash for the past few months instead. I don't think it's going to be pro because I don't think Apple wants the performance trade off of a reasoning model in in private Coud compute. alsoso I don't think they have I mean, if Google is giving them the servers to run it sure, I think that Apple would just prefer a speedier model that's going to be way smarter than Siri anyway. So yeah, that's something I wanted to talk to you about a little bit and just interrupt for a second, which is, I mean I know that The new I would say that the The reaction to Gemini three point five has been pretty muted in that it's not considered to the standards of either open AIs models or anthropics. On the other hand Apple will will be going so far by adopting Gemini, even if it's not the equivalent of you know, Opus or Jet GPT five five, it's still going to be remarkably better, I would think than anything that we have today. Yeah I mean, even Gemini two point five was going to be better than Siri today. So I think three point one flash could be a really nice improvement I am curious to see what they do for the promised features of Apple intelligence from two years ago when it comes to app intense, third party apps and Siri being able to context and perform actions. across the spectrum of compatible third party apps I would not be shocked to hear that to make it work with a new model scenes The situation has changed right compared to two years ago. And I think given also how the rest of the industry has evolved. I wouldn't be shocked to see some kind of MCP bridge in between the on device model, the apps and the cloud model based on Gemini, I don't think I would be I would be very surprised in a positive way if there's no MCP translation layer at all that actually Apple was able to get the Gemini model to understand app intents, the whole schema, the domains, all the things all the things that sort of they announced two years ago If it is MCP, I think it would almost by definition have to be private cloud compute given the context window, don't you think? Eespecially if someone has a lot of apps on their phone. I mean, it depends how they're architecting this, I suppose. I mean, in general, I just expect that the on device model will not be able to orchestrate too many tools at once because it will not be that smart I think it's going to be a small model. I do not expect. Have you ever tried running a small model like a small quen or small? I' tried to give it to give it tools, it. Yeah I've tried tri I've tried building things with Apple's on device model too. And it's almost unusable. I'm always impressed when a developer comes up with a clever solution for the on device model that Apple offers because it's very hard to use just because four thousand is not a very very good even But even if you set aside the context window, even if you set aside that no, I know they can't do tool call either, right? Right. Even if you set aside the context window and you say, well, well the context window doesn't matter because we're loading MCP dynamically now, which is Well what the rest of the industry is doing. You just don't have that smart model to orchestrate dynamic tool calling U And so if only because of that, I think it's going to be the cloud model. dealing with the orchestration and dealing with like when I mentioned orchestration, if the user says can you remind me when my next scheduled meeting with Johnn is and also added to my task manager? That's a whole bunch of information that you that you gota sort of organize and make sense of Where is the user storing the events? Who's John? When's this meeting? what task manager are they are they? What are the actions that app? What are the actions? What's the schema like? And I don't think an on device model can do that. So it's all I think it's going to all be based on the cloud model. and the MCP bridge is likely necessary because Gemini knows how to speak MCP. It doesn't know how to speak up intense unless worked a miracle in the past six months and they set up a whole system for Gemini to understand whatever schema they plan on rolling out for Appingtent. Let me ask you this. if it is MCP based, do you think Apple would open it up to other providers potentially No I think they will, I think I think the extensions that we have seen rumored in Bloomberg like that you will be able to connect your cld account. those will probably just be for having additional sources in Siri Yeah likeike the extensions basically, but I expect without tool be able to call tools or anything like that. right U I mean, happy to be proven wrong, obviously. I don't but from the sound of it, it doesn't seem like No making this series like a like a provider agagnostic. type of thing There's going to be the new series on a all' be based on Apple intelligence, Apple intntelligence has I don't think its it's even going to be branded as Gemini anything. It's just going to be Apple intntelligence and Apple Intelligence is able to get your contacts, get your tools, perform actions, on screen awareness Claude the Ch team up forb even grock. just going to be extensions and for web search and general world knowledge. Maybe somebody will be able to reverse engineer it and figure out how they connect together, you know? I mean Probably, but if it's the type of thing where the MCP brridge needs a private entitlement for it to work You're not going to be able to do anything on iOS and iPodS at least.. Maybe you can reverse engineer some of it on MacOS The problem I think, it's that they're going to make this MCP bridge that is cryptographically signed to your iCloud account on private Cloud compute. And so that's pretty much locked yeah. so but That's sort of the big picture of what I think is gonna to happen. I think they will shape what they promised two years ago but in a slightly different way from what was announced. Yeah, yeah. we should talk a little bit about shortcuts too because On the one hand, I think that the Sortcuts app as we know it today is I mean, it's not I don't think it'll go away, but I think it's going to take something of a backseat, But that doesn't mean that Shortcuts doesn't have a big role to play as part of what's announced at this WWDC because the rumors are that you're going to be able to use natural language. to create shortcuts, very much like what you did with shortcuts playlground Uuh And I know you have some thoughts about what the shape of that might be, but to me, it's very there are two things I'd like from shortcuts. One isd like I'd love to be able to do the national language stuff because I thinkre you're right that A lot of what you had to reverse engineer, I mean, obviously, Apple has hooks deep into its own system. It probably can do it a little more reliably, even though Shortcuts playlayground is very reliable as it is U But the other thing I would like Apple to do is really even though maybe shortcuts is going to take a little bit of a backseat to using it via Siri I would love to see Um them spend some time and just tighten up the app itself because I find that the app is still It got a lot of it's got it's just got a lot of bugs in it still that things just get goes sideways in the shortcuts app all the time. And I'd love to see all that stuff kind of like cleaned up a little bit Yeah, um I don't think Apple cares particularly about the idea of letting more users discover the advanced shortcuts capabilities I think Apple easily sell though. the idea of I've setting up routines Simple shortcuts, simple things, just with natural language. I think they're going have their own shortcuts playground and I think it's going to be better than mine for those things. whether they change the underlying shortcuts syntax, We U less reliant on XML and maybe even proprietary, that's a possibility I think even if they stick with XML It's going to be more reliable than mine when it comes to connecting variables and filling in parameters because they can do some pretty aggressive And obviously, They have more resources than me when it comes to teaching a model how to deal with that I think is going to be better than mine for simple shortcuts. I think that's going to be their pitch that anything that you do routinely on your phone, you can now just set up a set up an automation which workuts Playground using it I don't think it's going to be as reliable as mine and as powerful as mine when it comes to the nerdy stuff that you can do with Shortc' Playground. I I don't think you will be able to open up the the Apple shortcuts playlayground or whatever it's going to be called I guess if it's called that, they're gonna send me a season and the seas or something But I don't think be you will be able to open the Apple version and say, hey, build me a shortcut that talks to to doist to API and performs or runs a script or something or runs a script on my Mac or runs SSH things on my Mac Mi, I think it's going to be great for third party up intents, which is something that I intentionally do not support because again, I don't have the resources, but I think I my theory is that there will still be a place for Sortut's playayground as a tool for power users and advanced users and I'm more than happy to take that slot. But I think Apple will try and sell it as oh now if you you know if every morning you wake up and you realize that you open the shutters and you turn on the kitchen light and you want to start your coffee machine remotely, now you can just type that sentence in and That will become a shortcaster thanks to Apple intntelligence. Right. That's bitch. Right. It'll be like send, you know, send so and so a text message when I leave my office to come home from work. you know, that kind of thing, which is is you could set that up pretty quickly manually, but if you're not interested in Getting into the guts of the Surecuts app, that could be done pretty simply with an LLM. This episode of A Stories is brought to you by Things from Cultured Code If you want to achieve a goal, you have to have a plan. And when it comes to making a plan, there's no better tool than the award winning things app The idea behind things is simple create a project for each of your goals thenen add steps to reach those goals. and finally, schedule when you want to work on them Each morning when you wake up, things has already prepared your list of to dos for the day J spend a few minutes reviewing the list Put your to dos in the order you plan to do them and get on with your day It doesn't matter which device you're on Your tooos sink through the cloud So they're always with you on your computer, in your pocket and on your wrist You can also connect your calendars to see your events, schedule to dos that automatically repeat write notes with Markdown and a whole lot more. But when it comes to things, what you'll love more than any single feature is the app's design It doesn't just look good. it feels good to use. It's a native app madeade to a very high standard of quality and a two time winner of the Apple Design Award So if you haven't tried things, you should check out the latest version. Head on over to the website things. app That's things. app Learn about all the great features and things. Download a free trial for the Mac And you can find it on the App store. Just search for things whatever it is you want to accomplish in life. thingsings can help you get there Try things today at things. app You won't regret it Aart, thanks to Ths for their support of the show. Let's let's kind of go through some more of the wishes. I want to One of the things that I want to see and this is a very developer oriented thing, but Right now, developers only have access to the on device model. There are some developers who have built in clever hooks to shortcuts in order to access private clloud compute as part of their apps that, you know, that requires a trip to shortcuts. or it's a little hacky basically. And I think What I'd like to see is developers have the ability to very much like Apple has done with a weather kit. is Have a way to use private clloud compomute maybe have some sort of free tier up to a certain amount of usage and then have it paid after that Because after all, I mean, there are costs associated with this server kind of stuff just like there is with Weatherkit. But I'd like it to what I'd really like to see is Apple to find a way from a business standpoint for developers to make the cost of that use of private cloud compute. something that's predictable and projectable so that if someone's building an app and they want to charge five dollars a month, they'll know in they'll have a sense in advance whether they're going to be able to cover that with their subscription. I want to I don't know exactly how that works, but I'd like to see one it opened up and two opened up in a way that makes it more usable than a lot of what developers have to deal with when they're you know, working with APIs from like openp AI or Anthropy. Do you think developers will want to switch away from clouud or Cx because of this If they give them the right business deal, I would think they would. O security, private cloud compute to predictability on cost makes them makes it you don't mean for development. Do you mean for AI? Oh no,, not for development. I mean for I'm talking about LLMs are incorporated into apps for. So like So like right. So like when SQL, for instance uses an LLM to figure out what to drop into SQL based on the article you're reading, give SQL a way to do that on private cloud compute and not just have to use the on device model. That's what I'm saying No, that makes sense My wish is for First of all, the new app intense thing to be real. Fally the ship So this idea that we're going to have, you know, over the past couple of years, you and I've been using LLMs that have external integrations First with MCPs, then with CLIs, this idea that you're bringing your favorite tools into the chatbot or into the coding agent That's extremely powerful, right To be able to automate Sark that we use for email or to find messages on Slack and stuff like that I really want these app intense framework to be the native on device equivalent of that. so that all the apps that I use I can also use an LLM and search insightspark and search in slack and search in my calendar in my task manager. That I think could be an extremely powerful concept. but I don't want it to be limited to voice I really hope that this shifts as a way for you to ask Siri with your voice questions like the famous Apple commercial that they pulled when Apple intntelligence didn't ship, but also to have chat bot interactions with them I want to be able to text What's the latest draft in my working copy I want to be able to text it and to get a text response with with a visual snippet of that application in the Si chatbot UI. So whatever they do, I hope it's real and I hope it's not limited to voice and I hope that it It's multimodal in that sense, right? That I can use those apps However, I'm talking or speaking to Siri And then it deep links back into the app that it's giving information about, right? So you can easily go right in there and start working on working copy or whatever whatever you're doing. Yeah, I think's that's exactly what I want. I think that You know, that ad is kind of instructive. I mean, it was you know, it was at a partyence and, you know, the voice commands were being used. and that makes for a good commercial, but I think If you're at a party, most people are going to want to they're going to be typing. You're going be typing. you're not going to be muttering into your phone You know, who's the guy across the room who I met last week that I think that that's much more likely to be something that you would type in real life. All right, Federico, hereere's my big one I want well I'm going to I'm going to give you two And they're both related to the iPad. One we've already talked about, which is screen sharing. There's a lot of reasons to have screen sharing to allow you to check on Macs that are off doing things for you with LLMs. The other is, I think it's time to bring terminal to both the phone and the iPad We didn't talk about that in our IOS I was immagine clock code on an iPad. Yeah, and I think that you know there are a lot of good apps out there for connecting back to a Mac using a terminal style iOS or iPadOS app. I've tried a lot of them. and there's always a certain amount of friction because they are not system ms And I think if Apple were to do this m sorry to say it would probably kill an entire category that has been very active in the last two years. However, I think they it would kill it that category because it could be so much better than what exists Yeah, I mean, look, I would be so happy to see that, obviously It would need to happen with a terminal sandbox. Yes And in the sandbox, you can install Python, you can install Ruby, you can install nextJS, you can spin up a local web server in the sandbox, right? you wouldn't be able to use things like playwright or claoding Chrome because obviously you canot install Chrome in the sandbox, but it would need to be it would be a pretty serious effort to have a virtualiz handandbox where you can install whatever you want And whatever you want probably I'm guessing would give you a local user space, right? So the equivalent top Be you wouldn't have a desktop. you wouldn't have a downloads fold. No, but I would like to have the sandbox environment you're talking about, but also with hooks back into a Mac if you want them so that you can have things happening on your Mac and then securely passed back to your Right So it would need to have internet access, right because it would need to talk externally whatever comes back is sandbox to the terminal location I mean With enough money and time and skilled engineers, everything is doable these days It's only a matter of politics and priorities and strategy, right? I would love to see it I don't think it's going to happen this year. I don't I would love to see it No, I don't either. but it I mean, I've been trying a bunch more of these and every time I use one, I feel like developers are really get inching closer and closer to really good experiences on IOS and iPad OS with these terminal style apps. but There's always some friction. There's always something going on makes a little hard W son What was the second one? You mentioned you had two big things. Oh, well, it was just the that was the screen sharing, which we already talked about in the IOS episode So when he comes okay When it comes to the natural language shortcuts creation I hope that in addition to create I hope that you will be able to create one off shortcuts But also personal automations And I would like to see some kind of new design for those. Yeah I think you should be able, for example, to And I'm thinking of the automations in Kodex for Mac, right? Yeah. In anyime in KodeX on the Mac You can open a regular chat and you can say, hey, can you pull up my automation for, I don't know rossit triage in my email And so I hope that you will be able to open the Si chat bot and say, hey, can we take a look at my automations? And it pulls up like a visual preview, Here's the automations that you have set up. And here is do you want to make changes to them And it would be nice, for example, to get a visual preview of the automation. likeike seven AM, turn on these lights and you request a change. and in line in the visual snippet, you see the change being made to the automation. So instead of seven AM, they turn on at seven thirty AM, for example Automation, setting up personal automations, routines, whatever you want to call them, they've always had a learning curve for regular users because most people don't think about automating things. Right But there's potential for a system to let you create them easily Perhaps even proactively suggest automations for you based on things that the system thinks you're doing on a regular schedule. I don't know U, but I think there's a story to tell. notot just about creating chortuts with natural language, but creating and managing your automations with natural language Yeah, I think that that's a really good idea. The other thing I kind of want to see in the Si app is a mix of models approach the way perplexity and notion do it, where you can you can let Siri kind of choose you can you can swap between what models you have connected, you can switch them or you can let the system pick whichever it thinks is the best I think that's a you know for the company that's not actually making most of the models that are going to be part of Apple intntelligence. that's a powerful direction to go with this stuff What else you got? U so I kind of would like Apple to make Codex for for Xcode like their own version of Codex, something that that's a developer. coding tool that is not XCOode. I mean I think XCe is a great app and I think it ought to survive. But I do think that There are a lot of people who are who have kind of graduated to first, you know coding inside the terminal And then move to these more graphical versions, which have access to the terminal in you know, access to various X code tools and other things. and I feel like That's the future for a lot of young developers is not so much being in traditional IDE, but being in some sort of codex or claoged code style. window they can where they can draw in a bunch of different tools Cam iterate really quick, real quickly that way I don't think they'll do it Yeah, Xcode feels increasingly notot with the times. especially since development is moving to a more agentic type of workflow where given instructions and You don't need you don't need to see the different sections of an IDE like Xcode, for example. You're just providing feedback, assigning roles to agents and subagents, then testing, opening a browser, opening a simulator, taking screenshots, taking screen recordings, passing them back the agent deals. I mean, I have built IS apps for myself, the agent can deal with things like provisioning profiles or setting up multiple simulators. like all of that leegacy is obviously necessary because you've got to distribute those apps somewhere But Maybe it's time for a leaner, more modern experience that that abstracts some of the visual complexity away Yeah that's kind of what I'm thinking of. I mean, it's not that I would get rid of all the parts of Xcode that exist today. It's just that I think the focus needs to change where it should be more of an orchestration of agents. And then when you do need to look at the code open up a panel that shows you the things that have changed. You know get like a dipping tool or you get just the raw code itself and you have a terminal that you can slide up I mean I think Kodex does a really nice job of all this better than Anhtherpic does and it's You know, it's a powerful tool. I mean, it would be really cool to see something like that coming and built on with you know, from Apple Uh, what else do we have Let's see see Oh yeah, additional domains for App intents. I think When Apple announced Apple intntelligence two years ago, there was what In Domains But only some of them shipp in initially in September. Sure, never happened None of it ever happened. I'm a little surprised they didn't add to these last year to be honest because there's whole whole big air. Why would you add to them if you're not using that? Well, I know. but inastructure anymore. People could still start building for them. I mean they are doing magic. Can you imagine if they move away from I' All of that for something new and developers did the work for something that never shipped chang it again. Yeah, it's possible. It's possible. I mean,, you know, there are areas like I don't think, you know like health and fitness is covered or media like, you know categorizing media on your device like music or video or whatever. So yeah, I agree. I think that that should be greatly expanded because I think it's going to be one of my concerns about app andents is that people have forgotten just how limited they were initially And even if now the plumbing is in place to use them, there's going to be some frustration over where the boundaries lie with App ands. Yeah You have one in your list that I think we should discuss. The idea of integrating app intense across multiple apps Yes. What did you have in mind? So I I mean, and this is one that I actually don't think will happen, but Right now the way app and tents are designed to work is that these are actions can do that now the OS can call on to do them. in an automated way What I'd like to do is have the ability for one developer to call another developer's actions in their apps, make these like component parts that can be remixed by third party developers. What would that be like, you know, maybe maybe the readwise could call goodlinks, pull URL out of goodlinks and then display it using Readwise Readers, you know parsing engine. That kind of thing. It would have to be the sort of thing that there would have to be some sort of system of consent, I think, for that for it to work from an economic standpoint, but I think, especially for Iie developers, it's a tight community and I think There are places where doing that sort of thing wouldn't be could complement multiple apps without feeling like someone was taking someone's someone's work and use it as their own. So I'm definitely thinking of this as a collaborative tool, not so much as a, you know just grab whatever is available on someone's device and try to use it Yeah, the issue is obviously sandboxing again because like that is absolutely They can come up with some kind of structured data format extensions that run isolated from each other And maybe there's even the on device classifier in the middle that says, is this dangerous for the next st to receive or not? Like we've seen those type of things, like U, this is tangentially related to what you mentioned, but this idea of You have two LLMs, one big model in the cloud, one running locally, the one running locally is sort of evaluating data. for the risk of prompt injection and stuff like that Simon Wilson theorized this approach a couple of years ago. Google of all companies has a white paper on this system called Camel, I think. I think that's what it's called Yeah Wh they're using this dual LLM approach with one simply acting as as a almost as as a traic traffic light saying, Is this good to go? Th go. Is this dangerous? Stop Yeah. so I don't know mayaybe like that could help. Think of it as sort of an LLM based gatekeeper thing where there's only certain kinds of data that can be passed and they need to be evaluated as they pass the system. You know, URL's, texts, images, that sort of thing. And yeah, there are, I mean, obviously there are privacy and security risks there. so that would have to be addressed along the way I think we can wrap up the main show here, Jn Yeah, I think we can. you have a you have A Sties splus. Yeah. I want to talk a little bit about research and how we're going we're approaching those kind of things for WWDC. I've I think I may I think I may be on to something. it's still too early to be sure because it hasn't been battle tested yet, but I know you and I talked about this a little bit recently on Max Stories Unwind, but I'm a lot further along than when I was when we when we spoke about that originally, so I want to kind of give folks an update and talk a little bit about kind of the structure of what I'm trying to build And then we'll know after WWC whether it actually worked or not Okay. All right, Federico, well, that's it for today's episode of A Stories. I want to thank our sponsor for this episode. That's Things by Cultured Code. You can find the two of us over at macstories. net and of course, we are on social media where Federico is at Fitici. That's VI TI CCI and I'm at John Vortes. JOH N VWO R. HWS. Talk to you next week at Eederica. To John.