Evaluating Quality and Industry Impact

NPR . Think about some of the Western world's most famously prolific writers. You have horror novelist Stephen King. He's written more than sixty bestselling books and hundreds of short stories. There's also sci fi author Isaac Asimov, who produced more than five hundred fiction and non fiction books, and also English romance novelist Barbara Kartland, who wrote more than seven hundred books during her career. So much swooning. But today, large language models or LLMs can match that kind of output in a single year. The number of new book releases on Amazon almost tripled between twenty twenty two and twenty twenty five. That's an increase that can be attributed to LLMs. But are these AI written books any good ? This is the indicator for Planet of Money. I'm Adrienne Ma. And I'm Waylon Wong. Today on the show, we tackle this quality question through an economic lens. Plus we hear from a human editor at a niche publisher about how they're fighting the slop wars. Economists Imca Rymers and Joel Walf l have worked together for more than fifteen years . Imca's at Cornell University and Joel's at the University of Minnesota. Together they've studied how cultural products like music and books have gone digital. And sometimes this has involved some firsthand experience, as IMCA explains. I tried out to see how easy it is to self publish a book, not to try to sell any units, to be very clear, to just say , Oh yeah, I self published a book. Please don't buy it. What was your book about? Oh it was a half page about my day. It was nothing. I feel like you're underselling it, but okay. I really am not . Did you get two five star reviews? Yes, because two people did not listen to my plea not to buy it. Well, it sounds like Imc is part of this wave of self publishing because the number of self published books skyrocketed in two thousand eight following the launch of Amazon's Kindle E reader. Then, twenty twenty two brought another big disruption, the widespread availability of large language models like Chati Bit. Imco and Joel wanted to see if this new technology was also leading to lots more books being published, and if there was an explosion in AI generated books, were they any good? We'll come back to the issue of good in a bit, but first, Imco and Joel needed to figure out whether LLMs were driving a big increase in eBooks. They focused on Amazon, which has two thirds of the ebook market , and they figured out the number of new ebooks going on sale each month. Before the release of Chat GPT, it was about one hundred thousand books per month. And then essentially lockstep with the increase in the use of chat GPT, we see it increasing by later mid twenty twenty five to three hundred thousand new book s released per month. That is a tripling in the number of e books for sale. This includes all sorts of books, including those published in the traditional ways, as long as they have an e book manifestation, and of course every,one always releases an e book, but most of these books are self published works published through Amazon. The next step was determining how many of these books were written by AI. So Joelanimka ran a sample of around fifty thousand titles through AI detection software. What was really interesting there is that the share of books determined to be written by AI really matched the growth rate that we saw in the new books being created almost one for one . It was really just something . In other words, this boom in new ebooks could be attributed to AI . So now this brings us to the meaty question in Joel and Emcas research. How good are these books? And the word good here can mean so many different things. Joel explained how an economist like him interprets it. What we tend to mean by good when we talk about this stuff is appealing to some people, some people would buy it. So it's not the same as the good that a cultural critic would say. But fifty shades of gray, by this definition , the economic definition is undoubtedly a very good product because lots of people bought it. I'm gonna go pick up a copy of fifty chades of gray and see if you gave a blur for the back that says like this book is good economists like this book . No. Yeah, the economists understand that people like this. Yes. That's I think the blurb we would give it. This is a valuable product. Imco and Joel used star ratings and public Amazon sales rankings to measure whether people were buying these new AI books. And the results were clear . AI books had fewer ratings than human written ones. The average star rating for AI books was worse, and their sales rank was lower. By all those measures, the AI books are less useful or in some sense worse than the non AI books. In other words, they were not good . So Joel and Imcca say their research suggests that human authors don't have to worry, at least for now, about being displ aced by AI. Another interesting thing is there doesn't seem to be breakout hits emerging from the AI slush pile. Jul points out that, you know, Justin Bieber first got discovered on YouTube and fifty Sadhes ofay Gr, remember, a good book had its origins as Twilight Fanfiction published on the internet. These are artists who made it through, came out of left field as it were, and were made possible by digitization. AI doesn't seem to be like that. It's giving us more products, but they're not products all the way across the distribution. It's a whole bunch more near the middle . But having said that, everything is pretty new, so AI changes every day. And so maybe future AI will be more like regular digitization and less like big piles of middling stuff. Even these piles of middling stuff though have caught the attention of people like Jeremy Tar. He's the digital editorial director at Photos Travel. What was your last trip? My last trip was a road trip from Lake Como to the French Riviera. That sounds so glamorous. It was very glamorous. No need to rub it in Jeremy. So Potors has been publishing travel guides for more than eight decades and travel is one of the categories where there's been an especially big increase in AI generated eBooks, according to IMCA and Joel's research. Jeremy at Foders says he's skeptical of many of the new travel guide books he's seen online. If you were to see just one or two of them, it'd be, you know, somebody's just entrepreneur ial and they want to make a guidebook, which is that's great . But you then look at the author and the author has published a hundred of them just all over the world in a very short period of time. And knowing how much work and effort goes into producing a guidebook, whether it's one hundred and sixty pages or five hundred pages, it's impossible . Jeremy says he doesn't see AI travel books as a threat, but he does know that people are relying on Chat GPT and similar tools to plan their trips. So in July, Photos is launching a new AI product Eugene, named after late company founder Eugene Foder. It's a chat bot trained on Foder's guidebooks and digital content and it will give travel recommendations and help make itineraries. I think human backed AI is the best kind of AI you can get. It's like reading an article, reading the guidebook. I mean, there is so much information on any given city and it can take a really long time to plan something. So AI helps speed that up, it helps organize it. It's just a new way of finding the information. Jeremy says some of the large language models out there have already scraped photos material without the company's permission. Building a proprietary AI tool means the company can make money from this new technology, and it can control everything from the information it feeds Eugene to how the chatbot sounds. Eugene is is friendly , but no nonsense . I think a lot of AI is fairly sycophantic and we wanted to make an AI that is opinionated. If you say, should I go here or here, they will give you an honest assessment. I can't wait to have my travel plans totally crushed by Eugene the chatbot. That's a terrible idea. Only a dumb person would go to this museum. Okay , you say so whatever you say . This episode was produced by Julia Richie with Engineering by Cracey Lee, aspect checked by Sarah Wattas. Keeping Cannon is the show's editor and the indicator is a production of NPR.