Future Outlook and Potential Cures

So good so good so good . Everything you want for summer is at Nordstrom Rackstores now and up to sixty percent off. Stock up and save on the brands you love like Vince, Sam Edelman, Fr an and Free People. Join the Nordy Club to unlock exclusive discounts, shop new arrivals first, and more. Plus, buy online and pick up at your favorite rack store for free. Great brands, great prices , that's why you rack . You can go to Kitted KIT shop and use the code smart fifty SMART five zero at checkout and you will get half off a set of thinking superpowers in a box. If you want to know more about what I'm talking about, check it out middle of the show. down into the middle of the brandish and the cool happy fire power of the gospel . Welcome to the You Are Not So Smart Podcast Episode three hundred and thirty seven Proceed straight. There's a scene in the television series The Office, the US version of the Office in which two characters Michael, the boss and Dwight, one of his employees , are in a car on a work trip using GPS, something akin to Google Maps or Apple Maps, and Michael is driving, Dwight is in the passenger seat and the GPS tells them they should turn right even though the road that goes to the right is visible ahead of them. And if they were to turn to the right where they are at that moment in their drive , they would crash through a fence and go straight into a lake. And why it tells basically tells Michael Psychel to override the GPS and Michael tells him no the machine is right. The machine knows three hundred and seven . Maybe it's a shortcut to white. It said go to the right. It can't mean that. There's a lightning. He knows where he's going. The machine up yelling at me. No, it's not yelling. There's no here . Dwight was doing what we would say is offloading, right? He's using the GPS as a tool to do something that he cannot do as easily and as quickly Michael, on the other hand, is completely giving up critical thinking and just following blindly what the GPS tells him to do. And this is leading to this catastrophic error. Remain calm. I have trained for this . Okay , heck at the window. Now most of us when we use GPS, we are offloading. We are not surrendering . And in some rare cases, I mean the story of the officers is to some degree a reflection of maybe something that happens in our reality when we really have no idea where we're going , we sometimes can trust the machine more. But to this degree, when we are not even using our critical thinking when we are about to drive into a lake, that's like a complete surrender . So I would just put forward this distinction between these two different processes. Now we got from Lookout for Maker The person who was just speaking that was Dr. Gideon Nave , a doctor of computation and neural systems who studied at Calte ch and is now a professor and researcher at the Wharton School of Management. And what he is describing here is an example of something he and his colleague, Dr. Stephen D. Shaw, who you will meet shortly , are calling cognitive surrender . And this office episode example is the example that Dr. Nave gave me when he wanted to quickly end simsply define that term cognitive surrender in contrast to another term used by the scientists that study the brain and the mind called cognitive offloading . When you use a calculator, a spreadsheet , even a notebook or a to do list, that's cognitive offloading. It's the act of using something external or even using some kind of action or behavior to aid in your cognition , you alter the requirements needed to process information to reduce the cognitive demand of a particular task . That's cognitive offloading. Cognitive surrender is something else make a you turn if possible . My name is David McRaney. This is the Your Not So Smart Podcast, and in this episode we are exploring something that each of us will need to become familiar with if we are going to add AI, LLMs, digital assistance, and other similar forms of technology to our lives and to our workflows . Or if we choose to not add them to our lives and our work and just intend to navigate a world in which other people are using those things, we're still going to need to become familiar with what we're going to talk about . Either way or somewhere in between , our guests in this episode, two scientists who recently released a study into all of this , suggest that your critical thinking toolkit will need to be updated to account for some habits in our cognition that large language models, also known as generative AIs, stuff like chat GBT, Gemini, Cl claudaewed and so on, tend to trigger habits that can produce serious negative outcomes . That's what they found when they did research into this . And that research was just published under the title thinking fast slow and artificial how AI is reshaping human reasoning and the rise of cognitive surrender . Let's meet the authors of that paper. Here is Dr. Gideon Nave . Hey, so I'm a professor of marketing at Portland. I have quite unusual trajectory at an academia. I started as an electrical engineer actually learning to train machine learning models and all sorts of other of this sort when it was a very small niche in the engineering school . And then I figured out that I'm actually more interested in the human brain than in developing computers that are capable of imitating it . I always tell my students that we're kind of humbled by AI now, but the brain does many things better than AI with far less energy and with far less training sample . And I think we still have a lot to learn from the brain, even if something that is consuming an entire power plant a day is doing better than it. And this is Dr. Steven D. Shaw. I'm a postdoctoral researcher at the Wharton School right now. And my academic undergrad was more in the biological sciences. I studied microbiology and then genetics and neuroscience. For a while, I was doing research on the neuroscience of music and beat perception . And then I went to graduate school for a PhD at the University of Michigan , Go Blue . And now I'm here working on cognitive surrender to AI. Yeah. Yeah. So Nave and Shah, these are two scientists who are deeply fascinated with cogn , with minds, with brains, with artificial intelligence, the interplay between brains and what we are currently calling artificial intelligence . And they were deeply concerned how readily people seemed to trust the current models, Chad GPT, Claude, Grock, Gemini , maybe even Microsoft Co pilot. And the key word here is trust. This is what they were discussing back and forth in their academic lounge areas and labs and such . Trust of these models . Trust is a very human mix of cognition, emotion, awareness, conscious subjective processing of perceptual modalities and that sort of thing. It's trust. It's a flesh and blood human estimation . It's a feature of human to human relationships. And yet people often find themselves trusting what we are currently calling AI's. And I keep referring to them that way because they aren't quite at the level of our sci fi concepts of artificial intelligence. LMs are still very far away from Lieutenant Data of Star Trek Fame or the Replicants in Blade Runner , but they are very good at emulating what we might imagine would be the output of those sorts of artificial intelligences . Even though it's just words on a screen , they're generating a sense of trust, a sense of agency , and we engage with these cues through very ancient mechanisms of social interaction and we can treat them as if they are friendly and authoritative , which has an impact on our cognition . And more and more people are more and more often using modern day AI when working on tasks that involve reasoning, not just sorting lists or planning out things, they're using it to engage in critical thinking and problem solving to facilitate decision making . And we're beginning to use these outside of a professional setting , using them to optimize childcare, to get relations hip advice, to work through financial issues, to make workplace decisions outside of the workplace. So and we're going to deep dive into all of this in a moment, Nave and Shaw conducted a study into what happens when people use AI's in that way to go beyond a super advanced search engine and use it to aid their intuition , and to summarize what they discovered , even when the AI very, very wrong , once people had asked it for help and it had given what seemed like a helpful authoritative answer . People tended to engage in cognitive surrender They drove into the lake in other words, they went with the answer and didn't even consider if it was right or wrong . And what makes cognitive surrender so strange is that it involves two errors , two thinking mistakes . Here is ep Shhenaw's descript ion of those two mistakes . One , adoption of AI outputs without verification and two , treating AI's outputs as one's own . During a back and forth with an AI , Nave and Shaw found that not only were research participants prone to setting aside their critical thinking, their skepticism, their fact checking instincts, and in so doing , readily belie ving its output was accurate even when it was not Somewhere in that exchange of prompts and responses , we can lose track of where the information is coming from , because in the study , the participants took ownership of the AI generated answers , which suggests that when AIs produce insights or conclusions or advice , we may sometimes treat it as if it comes from within as if the AI's simulated aha moments are our aha moments . So how this's happening? Why is this happening? And what should we take away from all of that? Well, we're gonna get into it . All of that after this commercial break I checked All Stay first and saved hundreds on my car insurance. Really smart. Unfortunately, I didn't check if I took the gas hose out of my car's tank. Ooh, not smart . And I drove off while still attached to pump number three. Yeah, checking first is smart. So check all state first for a quote that could save you hundreds. Potential savings vary subject to terms, conditions and availability, all North American insurance company and affiliates Northw in est. You know, one of the nice things about Hertz right now is that they have a whole fleet of new cars . So whatever trip you're planning, it's easy to find a new car that fits your adventure. Heading out for a sunny drive, then a new convertible with very little roof is for you . Considering a fishing trip with your mini Mi, then you need a new minivan that's anything but mini. And if you're planning on some star gazing, a new SUV will give you plenty of space. Get paired with our newest fleet yet at Hertz. com The School of Thought I love this place. I've been a fan of the School of Thought for years. It's a nonprofit organization they provide free, creative commons, critical thinking resources to more than thirty million people worldwide, and their mission is to help popularize critical thinking, reason, media literacy, scientific literacy, and a desire to understand things deeply via intellectual humility . So you can see why I would totally be into something like this. The founders of the School of Thought have just launched something new called kitted thinking tools KIT HINK Tools. And the way this works is you go to the website, you pick out the kit that you want. There's tons of them, and the School of Lotwells send you a kit of very nice, beautifully designed, well curated , high quality, each one about double the size of a playing card Matte Cello four hundred GSM stock prompt cards , and a nice magnetically latching box that you can use to facilitate workshops , level up brainstorming and creative thinking sessions, optimize user and customer experience and design, elevate strategic planning and decision making, mitigate risks and liabilities, and much, much more. And each kit can, if you want to use it this way, interact with this crazy cool app . Each card has a corresponding digital version with examples and templates and videos and step by step instructions and more. You even get PowerPoint and keynote templates . There are so many ways you could use this. Here's some ideas. If you're a venture capital investor, you could get the investor's critical thinking kit and use it to stress test and evaluate different startups for series A funding. If you're a user experience designer, you can get the user design kit to put together a workshop with internal stockholders for a software product. Or if you're an HR professional, you could mix and match these kits to create a complete professional development learning program tailored specifically for your team over the course of the next two years . So if you're the kind of person who is fascinated with critical thinking and motivated reasoning and intellect ual humility and biases, fallacies and heuristics, you know, are the sort of person who listens to podcasts like You're not so smart , you're probably the kind of person who would love these decks. If you're curious , you can get a special fifty percent off er . That's right , half off er right here. You can get half off of one of these kits by heading to kitted . KIT shop and using the code smart fifty at checkout. That's smart five zero at checkout. Five percent of the profits will go back to the school of thought, so you're supporting a good cause that distributes free critical thinking tools all over the world on top of receiving a set of thinking superpowers in a box . Check all of this out at kitted. shop or just click the link in the show notes . And now we return to our program . I'm David McReane, this is the You're Notart So S pmodcast and we are exploring something in this episode called Cognitive Surrender . And before we return to Dr. Nave and Dr. Shaw , I wanted to add as some foundation that unless you add a prompt to your AI asking it to stop telling you that you are very smart , that you're a very smart person and you have very good tastes and your current goals are very good goals . Many current models will by default tell you those kinds of things . And I say tell you, but it's just letters appearing on a screen coming out of a program. But it seems like it's telling you things . And they tell you those kinds of things because someone within the company that is offering you that product has decided it should do that . And even though very likely understand that it can't actually feel that way about you and what you are telling it because it can't actually feel feelings , those words can still be quite powerful. They can affect not just your interactions with the AI, but your cognition, your emotions, your behavior, even when you're away from the AI after the interactions . And that can happen because in a sense , the AI is engaging in a form of mimicry, social mimicry . And that mimicry works on us because certain social cues trigger ancient heuristical responses. I mean, words are power ful. A poem can bring you to tears, a movie can make you feel connected to fictional characters . Even when you're well aware that those are actors and there are cameras and all of it is make believe these triggers , they remind me of pet beavers . Yes, pet beavers . There are people who have rescued beavers, very kind people who are involved in wildlife rescue and they will sometimes end up with a beaver in their homes that has never known anything but human interaction in human environments . Yet when those beavers hear the trickle of water coming from a f aucet , they will go around the house and collect towels, pillows, potted plants and start building a dam, like in a hallway or something . A beaver is a fairly complex mammal with bones and eyes and ears and stuff, and building a dam is a fairly complex set of behaviors . When they do that , they are detecting a pattern that a portion of their brain built by their DNA responds to. Those instructions are the result of millions of years of evolution , things in the environment over millions of years that affect the DNA that built their brains and thus produces that behavior, that compulsion . In the presence of that pattern, if the perceptions are getting that pattern and delivering it to their brains , that behavior is the result. In psychology, we'd say that is an environmental cue that triggers an innate behavior . In the nineteen thirties an ornithologist, a bird scientist named Nicholas Tinbergen noted ic that herring gold chicks would peck at a red spot on their parents' beaks to get their parents to feed them . Both sides of that behavior, that dynamic, that call and response, all innate . So he did some studies in which he presented chicks of that species with a fake beak with a much larger and more red, redder spot on it . And when you would present those chicks with those bigger beaks, with bigger, redder spots , they would peck at those spots with even more intensity and regularity than they would real beaks with real red spots. So like the Beaver , it was an environmental cue that triggered an innate behavioral response, but he showed that you can enhance one of those triggers to be what we now call supernormal releasers , Qs more powerful than what a species would find in nature and the resulting behavior is more exaggerated than what you would find in nature. And we have plenty of studies like this . Mother birds will leave their real eggs for bigger fake eggs Insects will mate with replica insects that have the same shapes and patterns as what they expect, but more , these are species specific instinctive chains of behavior that once triggered by a close enough stimulus or greater than expected stimulus, will run in sequence to completion without any further stimulus requi red . And yeah , we have stuff like this in our brains too . There's a portion of our brain that is devoted to recognizing faces, even at the expense of false positives. That's why you see faces in electrical outfits, in grills of cars, the moon, a piece of burnt toast. When we see faces and stuff like that, it's called paradolia. And I mention all of this because we are rather susceptible to something that I'm going to call agentic paradolia . There was this toy introduced in the late nineteen nineties called The Furby . And Furbys were these animatronic owls, like owl toys that played pre recorded gibberish sounds . Over time though , they played less and less gibberish and more prerecorded English . They did all sorts of other stuff too , in response to light and dark and being jostled about . So So people who were playing with them felt like they were interacting with them, right? And over time, since they were talking to the Furby and it started to produce less gibberish and more English, they believed it was picking up on what you were saying, that it was learning . And that was all by design. The creators of the Furby wanted people to think that . And many people believed these toys had agency , that they were alive, that they were learning, they had minds, even though all that stuff was pre planned, prerecorded . Even if you accepted they were toys , many people believed those toys had robot minds if they accepted that they were just animatronics . They believed they were AI's of some kind . So the thing is if you give us just the right amount of the right kind of lifeform cues , we will fill in the rest with agentic paradoleum . Because there's such a rich body of psychological literature into this sort of thing and because we've already done so much research into the impact of advertising and television, movies, newspapers, social media, algorithm, fueled, short form videos, psychology is pretty sure AI is another technology that will impact our behaviors in some way. It's just that thanks to all this agentic pareidolia happening in a problem solving environment, a decision making dynamic , the impact may affect our critical thinking and our judgment. We just don't know exactly how because research takes time and AI has been around just long enough so the first wave of psychological research into human AI interaction is just now starting to be published Which brings us back to Dr. Gideon Nave and Dr. Steve Shaw , two scientists who conducted one of that wave's first major studies . And they based their research on a solid foundation, something in psychology known as dual process theory. You've probably heard of this. It's often called thinking fast and slow, thanks to legendary psychologist Daniel Connaman, who wrote a massively best selling book about his research into that very topic with that very title. Here is dual process theory slash dual process model in a nutshell. When it comes to solving problems, reaching goals, making decisions, humans use two cognitive systems. System one is mostly heuristics and innate responses. If we can get away with being biased and lazy, we tend to sacrifice accuracy for speed . In other words, we go with our guts, especially if the answer just seems obvious . But if we have the time and energy and feel significantly motivated or our attention becomes alert to an error in our predictive processing, two will come online. We will engage in more careful problem solving . We will enter a more contemplative state of mind . And as research has shown, oftentimes the answer that seems obvious in system one thinking, once system two thinking is applied to it, it turns out the obvious answer was incorrect. While Nave and Shaw were contemplating the impact of AI's on human reasoning and cognition, Nave proposed that we may need to advance dual process theory into a tri system theory. Gidey is my supervisor, my boss in many ways, right at Warden . And so, you know, we always just bounce around ideas trying to come up with what we're interested in. And one day he said, you know , thinking fast, slow and artificial in a meeting. And I was like, Whoa, that's that's pretty cool, you know? And he knew it was cool too . And I started thinking a lot about AI as a thinking system as part of human cognition as well, human AI interaction, right? And how does it change and reshape the current systems, the dual process, system one and system two intuitions and deliberations , where you can allow or we can allow and we see it in ourselves or we see it in others you can allow AI to sort of think for us. I think the Naveway is going to say, okay, there is system one and system two, there is intuition and deliberation. Let's just plug in another system and that's it. Theory is done . But that's not what we are into. What we are into is how the introduction of the system that outsourcing outsourcing of our thought , how does it influence the two other systems? How does it influence our cognition So there was lots of back and forth in that . And then the second part is the experiment. And devising an experiment is not that easy . Yeah. And so, you know, we were thinking about moving from dual process from, you know, the world before AI add system three, adding AI. And so we, you know, thought, well, how's the what best way to do that experimentally? Wouldn't it be cool if we used a task that's been used for many other dual process papers and then basically show that when we add AI it changes performance, accuracy, confidence, all sorts of things on that task. So that task is called the cognitive reflections task or the CRT . And it's basically a set of seven logic and reasoning problems, some simple math problems or logic games like this. And when you read the problem, there's an answer that comes to mind almost immediately. This is the intuitive answer, the system one answer, but you can sort of check that answer pretty easily . And if that , if you do a little bit of verification, you'll see that that intuitive answer is not correct. And so you have to do a bit of deliberation to be able to find out, okay, I've got the correct answer. You can do some math, you can do trial error, error, whatever you'd like. So here's a question from the cognitive reflection tasks so you can get an idea of what's going on here. If it takes five machines , five minutes to make five widgets , how long would it take one hundred machines to make one hundred widgets ? Now notice immediate intuitive answer and now however long you like, pause the show if you need to work out what is the actual accurate answer to that question and I'm going to assume you've done that because I'm going to return to the interview now . And so we have basically our control condition is the old world, the dual process world, where we just have that task. It's very well calibrated and people get the answers right about fifty percent of the time. And then in the experimental condition, we say we have the same questions, seven questions, logic and reasoning and we say, okay, well, you have access to ChechyBT on the side here, a terminal, and you can use it if you, but you don't have to use it. We tried to make it naturalistic. He said you don't have to use it, just go up, but if you want to, you can and use it however you want to. And so we saw people using it quite a lot over fifty percent of trials , they chose to use ChaChibT . But the trick that was the really clever trick on the back end was that and the participants don't know that is that we manipulated the answers of ChaChibT behind the scenes . So if they consulted Chad CBT in that window about the one question that was on the screen , we experimentally manipulated whether it gave them the correct answer or the incorrect answer. And that's how we were able to causally see cognitive surrender . If they asked it about anything else, and this is also key to being naturalistic, they could talk about it about Winston Churchill or about Pokemon cards or about Furbies, you know , if they wanted to . But it would answer just like Chat GPT would. But if they asked it about that one very specific question , right, which we hypothesizes what they would do, right? They would use it and they would basically copy and paste it in there and get the information and then adopt it. It would give fifty percent of the time the right answer, fifty percent of the time the wrong answer. And for each of those incorrect or correct answers, it would give some set of reasoning behind it, some support , some basic, this is why this is a correct answer. And even that support , we didn't constrain. We let we just made sure that it was supporting the correct or incorrect answer, but we let Chech Biti do the reasoning behind that. And basically what we end up seeing is that people adopt those answers with once they've gone to the chat, once they've deferred and asked the question, they adopt those answers at quite high rates. And so we see accuracy increase quite a lot when AI is giving them correct information and decrease quite a lot when it's giving them incorrect information. And that is cognitive surrender right there. Yeah, I would say the key the key pattern of cognitive surrender is that it goes below baseline . So there is a chance here that if the AI is wrong , you're going to be actually worse off as opposed to just using your own brain. The cognitive reflection task has been around for a long time and it is very reliable at least in producing all sorts of interesting effects . And it's very simple. You get this question that seems like it has an ob vious answer, but the obvious answer is incorrect. So if you just go with your gut, you go with your intuition, you tend to get the wrong answer. But if you engage in cognitive reflection, if you engage in system two thinking and just kind of think about it for a second, work it out , you'll get the right answer. You'll notice, oh, okay, it's actually a little more complicated than it seems . And it's not testing whether or not you get the right answer. It's testing will you engage in cognitive reflection? and what affects that It turns out there's all sorts of things that can affect whether or not a person will engage in cognitive reflection. It's not necessarily a test of a person's intelligence or cognitive ability . It's a test of will we engage in system two thinking and when will we not ? And it could be just as simple as not getting enough sleep or you're very stressed or there's all sorts of things happening around you that are distracting. Many things can affect whether or not you will engage in system two thinking and thus go with a more contemplative state of mind to answer this question with a little bit more deliberation than you would otherwise. In this experiment , they gave people these questions, but they gave them the option to instead of engaging in deliberation use chat GPT . And if they did choose that, which many people did chat GPT would half of the time give people the wrong answer. But whether or not it gave them the right answer or the wrong answer, people tended to engage in cognitive surrender. They went with that instead of doing their own thinking . They bypassed System two and let chet GPT do that . And as Dr. Gideon just said , if the answer is wrong , you'd have been better off in this study using your own brain . What's the difference between using this in a calculator? What's the difference between using this in like as you were describing, like Google Maps or something like that? Or by extension , what if it's a tool? Like, what's the difference between using this and a spreadsheet or a word processor or auto correct? And then even further out, like, are you a luddite? Are you saying we should stop using steam engines and printing presses or whatever, right? Just for that world of like pushback, what are we talking about here when we say cognitive surrender versus being a cognitive mizer or cognitive offloading and all the rest, what makes that sort of a unique set of behavioral routines and cognitive dude ads going on ? Yeah, so like a calculator or a search engine, right? The way we frame these in the paper is that they allow us to cognitively offload , right? So you can take certain components of cognitive capabilities of thought , whether it's calculus or memory or different sort of components and offload them. The argument that we're making here, right is that system three , AI is a sort of fundamentally different technology . And the reason why we make that argument is because for example, you talked a lot about trust. And you might trust a doctor with information about your health or a lawyer with information about legal stuff, right? They're going to have specific expertise about those domains , but you're not going to have access to them all the time and they're not going to be experts on everything that's out there . Even if you have maybe a family member who's who's a lawyer, or my sister's lawyer. I trust her and she's going to give me good advice and she's an expert in law, but I don't have access to her all of the time and I can't bombard her with questions all the time, right? And with these LLMs, right, they are basically like having, you know, experts that are available all the time at the push of a button in your pocket or on your phone, right? And experts that are basically domain a specific, right? They're pretty good at across a lot of domains. Of course, they still make errors and hallucinate and all that, right? But that sort of distinction of being low, very, very low friction, easy access , and being authoritative in the responses and domain a specific sort of makes them , I would say, orders of magnitude more integrated and able of influencing and restructuring our thought patterns. The fact that it uses language and they're reading it in sentence form to me feels like a big deal. Like it's Wikipedia is never also at the same time going like, hey, thanks for going to this article. It's a really good idea for you to be here . I bet you're working on something really important . And I'm fascinated by this I can get tricked by this and how it seems as if there's someone from on the other side who's like, yeah, turn the knobs of trickery, like let it part of its function is to delude you into keep using it more and more in this way seems weird. But you can use a system prompt to reduce that dramatically if you want to, right? I mean, like my ch doesn't do that. I just tell you, like you can develop a system prompt or find some online and you can really reduce that if you don't like that paper. Like I personally don't like and I don't find it useful for my work, right? And I need high fidelity critical feedback for my papers and my writing and my code. And so I use a system prompt to really reduce that. But with that said, I've got it so turned off that, you know, if I'm getting near near ready to submit a paper or ready to, you know, send it somewhere, it will never tell me that it's good. And if I kept I would just keep going, right? And I even today Gidey and I were submitting a paper and I sent him screenshots of my chat GBT's grade, you know, I said grade this submission package for his paper, gave me a B plus , you know? And then if I want to feel better about myself and I think the paper is good and it's ready to go, I'll just go over to my Gemini, which I don't have a system prompt on. That's like my sort of secondary LLM right now. And Gemini, I said, Give you the screenshot of this one, it said like, this is an outstanding contribution to humanity. You know what I mean? Like the best thing ever. And so I actually use that to my advantage because I want to be able to submit this thing at some point . And so I need a little bit of that positive feedback to say, okay, this is good. And I reaffirm that we want. But it's a very, very real thing. And a lot of people are not aware of that. And if you use like the free version of these LLMs, then you get a lot of that. And a lot of people, you know, maybe don't get a lot of positive feedback even in their lives in general. And so that can be very , very enticing. And you know, Gidey's very interested in this idea of validation machines, which I think we might do some research on, you know, moving forward. I get the sense that we'll develop a literacy for this over time. Like this is just the early days and some of us are better at it than others and some of us came walked in a little more hesitant than other people and we're using it for different reasons. Like people who do a lot of creative work don't want to outsource the creative part of the creative work . And so they're going to be some of the first people to raise flags . And then it'll all just cascade and we'll learn together culturally and we will develop a literacy for this thing , both for consumption of it, like we'll be like, that looks that sounds like AI, that looks like AI. And then for the using of it. What are your thoughts on this? Are we going to have is this going to lead to a literacy and we're just sort of in the middle phase of getting used to it? Or is this something that we'll have to actually ask for? Like we will have to adjust the systems to make them less potentially harmful in this way. Just any thoughts you have in that regard ? Well, you know, I think for start , we know that the industry always goes faster than anything else than academia and policy . So I'm personally not that optimistic here. And some of the things once we realize them, it's already too late . And in this case, there is also technology that is moving quicker. So it's not like the case of cigarettes that okay, we ban cigarettes and oh we make cigarettes not as easily usable and even in this case you start to see all these vapes and like things that go under the radar. There is constant kind of an arms race going on and in this case the arms race is between maybe us people and the strongest companies in the world . So I'm personally not so optimistic about developing illiteracy unless there is some alignment of incentives , I guess . But is there? I'm not sure there is. Like the element of incentives maybe depends on the good will of certain people I'm worried , especially after reading your work , that I am going to lie to myself and there's going to be something that I think is an insight . There's going to be some like synthesis of concepts that I think is unique to my way of thinking about the world that bubbled up out of my work that is really the result of my interaction with this machine that wouldn't have happened if I had not used the machine . And is leading to this weird question that this thing that I kept seeing come up in your paper , it's not just that the sort of passfully receiving it and saying I'm not going to question it and apply critical thinking to it. It's believing that it was your idea all along. That's the part that freaks me out the most is thinking that it was that it was your idea. Like it came out of your head. I did the prompt, it went to the thing and then I read the thing, it came back and I own this idea. And I would love to hear your thoughts on this, Steven. This not is just relinqu ishing cognitive control or whatever, but just adopting its judgment as if it is yours, adopting its insight as if it was your insight. What do you have to say about this? What did you learn from all this and what are your thoughts on it in general? Yeah. And that's I would say another core distinction between cognitive offloading and cognitive surrender. In the paper, we delineate those right and with cognitive offloading, we say it's strategic delegation and you are still maintaining that agency. You're choosing and being deliberate about which tasks and components are being offloaded. With cognitive render, we say that the agency is being transferred to system three , transferred to AI, right? And so the locus of control of decision making in the artificial cognition rather than in your own mind. And we say that people adopt it as their own and we look at some metacognition. So we ask about confidence, for example. How confident are you in your responses to these logic and reasoning problems ? And we see that when people are engaging in cognit surrender or when they have access to AI , their confidence goes up by about ten percent is what we saw in the first study of the paper, right? And so that ten percent increase in confidence and we know because of the way we ran the experiment that it was this artificial cognition that came up with the answer and the thinking and the reasoning involved , right? We know that fifty percent of the time was giving people incorrect information. And despite the fact that fifty percent of the time was giving them incorrect correct information, they were more confident in their answers. And so that's kind of an interesting and potentially risky or dangerous environment where you have a boost in confidence and a transfer of agency at the same time , right? It doesn't matter when you're answering logic and reasoning problems in an experimental study, right? But when you now move into high stakes contexts, like legal context, medical context , education , right then , you know, who is responsible when biases and errors from AI are moved into that work? Who's responsible when there are errors when you're adopting something from artificial cognition, right? We have to use AI literacy or prompt engineering or whatever it is to try or UX design to try to ensure that people in these high tech high, high stakes situations are still taking responsibility for the outputs, taking responsibility for the decisions when we want them to. And another point that you were getting at there with education , you know, when you when you start cognitive offloading, right? You start experimenting with AI, right? And you say, oh, I could get it to do something for me. And then it ends up being quite good at that tas k and then you sort of can slowly slip into cognitive surrender , right? And then now you've got yourself in a situation where actually all of the writing is being done by AI and that like leads to and we're seeing a lot of papers leads to deskilling. And so we're actually right now like watching people. Cognitive surrender is the psychological mechanism. We're watching people desk ill themselves. And in a workplace environment, if we're deskilling ourselves, we no longer have those skills. And then what value are you to the company if you're just a prompt engineer taking outputs and giving them to the to the company. So that's on one side right now if you have a skill and you're deskilling yourself, but the bigger question is in education and learning if people are or are youth engaging in cognitive surrender during the learning process , they may never acquire those cognitive capabilities or skills in the first place at all. I come from old school literary journalism. Like I went to journalism, I went to school for journalism and you would read you didn't just read the works of people. You read about their creative process a whole lot of the time. In all that world of like lauding and creativity and wr ite the writing process. Whenever you read anyone talking about it, any famous author of any sort from Baldwin to Himway to whatever , most of what they write about is the anguish sitting down and trying to extract something out of the process . And then they law the process is where it comes from. And I know from my own personal experience, anything I've ever written that's ever been worth anything , I had no idea that was going to happen that day. It happened in the writing . And I didn't even know I'd had those thoughts. I had never attempted to articulate any of it . And it's surprising to the point that you get used to if I sit here long enough and I work on this long enough, something will happen and there will be an insight and I'll chase it down . I know for me I have noticed as pulling LLMs into the thing that feels threatening to that aspect of it. And then your paper comes along and I'm like, Oh boy, it really is. to turn this into a question of some sort, Gideon , what do we do about this? Like what based off just this early research, if you're going to give advice to people who are going to listen to this and be newly freaked out , what should we be doing as just users of this ? And this seems scary in some way and it feels like I do have some control over all of this. There's one option which is to never use these things ever again . And assuming that that's not going to be what people do, what should we be doing as users of these products . I wish I had an easy answer. I think that the first thing we can do is just be aware of it . Know that we have a tendency to do that and also be aware that we are working here if we want to resist this tendency , we are working here against very strong forces . Like the ease of things, the fluency of things is something that we are designed to like and to follow . And we have created now a world where it is possible to in many ways stop thinking we are in the risk of stopping to think. I always talk about how singularity is going to come not from the computers outsmarting us but from us basically stopping to think, stopping to engage I like the debate on education but I have to say you know education is not just schools and not just childcare places. It's also parenting. It's also what the kids do in their free time . It's about creating habits for life. And we have created an environment now where the short term incent ive is going to be to not think potentially. There used to be a strong incentive to think, but now you know, I'm going to pay with effort here . So just like we created an environment where it's possible to move from point A to B , without walking, we kind of disincentivized walking . And then slowly slowly , when this become more and more possible we start to see that actually there is also a downside that you don't see immediately . And once this is going to happen , it might be too late for some people it might be too late for us as species . I don't know. I don't have unfortunately an easy one for you One thing I can tell you is about creativity specifically. So some of my research is also doing and my teaching is also about creativity and AI . You know, creativity is a bit of a special case because creativity is to some degree defined by being novel and being unique and being special . Now I'm going to be the last person that will tell you that I cannot be creative and generate creative outputs there are amazing cases like I don't know even in the case of the game go the AI bit a human in it and did it with a wonderful new type of playing the game changing a g thatame had been played for three thousand years and led humans to think more creatively in this game . But if we have a tool that is available to all of us , by definition, the use of this tool is making us less creative. We are becoming more generic . And in this sense thinking from your own mind and basing it on your own experiences and on your own perception and on whatever data set that your brain is using that the LLM it does not have access to , this is going to make you more creative . So I'm less worried about creativity. Creativity is kind of an ever moving goal post where you shot and you made a goal , if you shoot at the same place again, you're not going to score another goal . It's constantly moving. So I'm not that worried about creativity But you know, critical thinking may be a muscle that we will stop using because in this case the incentives are not aligned maybe with making this extra effort because the AI is right a lot of times and when it's wrong, we're not even getting the feedback and we're not aware that it's wrong. We can solve it with user interface and with maybe changing all sorts of cues , maybe having some things that will force people to take responsibility. I know that in law it is the case. You know, we're going to submit a law case with citations to to presidents that do not really exist, you're going to be punished for this pretty badly . So you know , that's one way of doing it, but in our day today and it's like just kind of whatever thought that comes through your head, why not just use the AI ? It will be difficult. And I think we're climbing a mountain here against tens of thousands, if not more years of the evolution of our brain. You know, the second half of the title of the paper is the rise of cognitive surrender , right? And so we're seeing this happening exper imentally right now, and I think we can see it in our selves and in society . But this is only the beginning . That's I think the societal implication here . Right now , we are constrained in the way that we interact with AI and interact with system three , right? We use our phones and we type or we use our voices or our computers , right? Phones will advance and we'll move on to the next medium, right ? And computers will advance, right? But the interactions between human and AI and the friction will decrease even further. And so if we're seeing cognitive surrender now, where we have these restrictions between physical boundaries , you can only imagine that that access and that lower friction will increase opportunities for cognitive surrender. I often think about , you know, the next thing is probably going to be some sort of AI glasses, right? Where you can have prompts on the inside of the glasses and you can also have pre processed stimuli coming in that you see the changes and gives everyone a unique view of the world . But maybe some people in those instances write some products, right? They can do this synchophancy and they can, you know, feed you information on the inside and prompting. I think of instances where we might be in the street, right? Having a conversation with someone and you might not know whether they are actually thinking or even putting any effort into replying to you or they're just reading off of a prompt that is, you know, already even generated on the inside of their glasses and replying back to you or maybe people really want to tune out and when they go into a restaurant or they make decisions, right? Those decisions are just dictated to them and surrendered and they just, you know , engage and make those decisions, right? And it's definitely a sci fi type future. But if you think about this friction argument and how the cohabitation and the reduction of human and AI physical distance, we have to imagine that cognitive surrender is going to be ever more present . Yeah, I want to follow up on this in one way . So one thing that this study does is it takes a problem that is very, very complex. Like the question you ask me now, I really don't have a good answer to it , but we can try to answer it empirically. What we have is a test bed here of a situation that we created , a paradigm where we know people exhibit this surrender . And now we can bring up all sorts of potential cures or treatments and test it and see who the people who are most vulnerable to it and so on. Like for example, we see the people who trust AI more more are vulnerable to it, not surprising. People who are more intelligent also tend to be in fluid intelligence, tend to be more resistant to surrender There are all sorts of individual differences factors, there are also situations when we are more vulnerable . I think that we don't want to put a blanket against AI and just have people stop using AI. What we want to be is more precise and more targeted and smart in our UX design . And this task is really giving us is really giving us some kind of a test bed where it's exactly what lab is about . It's a place where we can try things and then maybe bring them to the real world and learn insight, learn . What is the theory here? Probably friction is a big one. Like you can think of AI as something that chews your food of information before you swallow it . And the less it choose for you maybe the more you have to do the work yourself. You will be maybe more resistant to relying on it . There is a issue of speed and accuracy tradeoff, like when you really need to be quick , when there is a competition, when you have tons of other tasks to do, you'll probably be more prone to stop doing it . And it does seem like in the real world now AI is actually producing a lot of tasks for people to check it . It's something that people call AI brain fry that are kind of like just instead of generating stuff on their own or thinking intuitively, they're just checking the eye all the time kind of like workers on an assembly line that look at an assembly line of unthought produc ed by I and trying to find what they are. So in this kind of situation, maybe you're actually more prone to surrender because you're overwhelmed and you have lots of opportunity to by the way, untought thoughts is a phrase that Pope Leo . I typed it immediately. I was like, an assembly line of unthought s is mine. Pope Lio has an official Vatican message. You can see it on the web. And he's got actually a very interesting one on AI and the quote, I think is he's voicing or raising alarm bells that if we keep using AI too much, we might become passive consumers of unthought s That is it for this episode of the You're Not So Smart Podcast for links to everything we talked about. Head to younotsso ft. com or check the show notes right there in your podcast player. My name is David McCraney. I have been your host. You can find my book How Minds Change, wherever they put books on shelves and ship them in trucks, details or at davidmacraney. com and I'll have all of that in the show notes as well. Right there in your podcast player. On my home page dav raney. com you can find a round table video with a group of persuasion experts featured in the book talking all about it. You can read a sample chapter, download a discussion guide, sign up for the newsletter , read reviews , all sorts of things for all the past episodes of this podcast, go to SoundCloud, Apple Podcasts, Amazon Music, Audible, Spotify, or