Using ChatGPT for Book Research? Take Exceeding Care

Today’s post is by author and book coach Susanne Dunlap (@susanne_dunlap).

WARNING: Research Rabbit Hole Ahead

I do a lot of research for my historical novels, so I’m interested in knowing about anything that might make that process faster or easier for me. And honestly, I’m curious about ChatGPT in general. So I thought I’d put the app to the test against my usual internet searches and see how it measured up.

Before I get into what I discovered in the course of my experimentation, it’s worth noting that the ChatGPT search window says, “Free Research Preview. ChatGPT may produce inaccurate information about people, places, or facts.” This is important to bear in mind in any research—even in primary sources, where you can easily find conflicting information.

I started by asking questions related to my work in progress.

My dual-timeline manuscript takes place in a mill town in Maine at two different times during the 20th century. One key location is the Biddeford Opera House, which was opened in the late 19th century and still exists as Biddeford City Theatre.

I had so many questions about what the venue was like and what shows were mounted there during my two timelines, but I decided to start with the basics.

Question: What is the History of the Biddeford Opera House in Maine?

Google turned up “About 1,470,000 results (0.61 seconds).” The first five results were for Biddeford City Theatre, listing its programs and so on. At the bottom of the page, there were links to some historical information on the Maine Memory Network (great site). I clicked over to that site and found some fascinating pictures, but no history.

Not very impressed with these results, I went over to the conversation starting point in ChatGPT and typed the same question.

In seconds, it came up with a pretty decent potted history that mentioned Sarah Bernhardt, Enrico Caruso, and John Philip Sousa performed in the opera house—something I hadn’t seen in my previous limited research.

Encouraged by this, I decided to get still more specific and ask, “When did Caruso perform at the Biddeford Opera House?” To which ChatGPT answered:

Enrico Caruso, one of the most famous opera singers of all time, performed at the Biddeford Opera House in Biddeford, Maine, on October 22, 1909. Caruso was at the peak of his career at that time, having achieved international fame for his powerful tenor voice and his performances in operas such as “La Bohème,” “Carmen,” and “Pagliacci.”

I tried posing the same question in Google and was still served a dizzying array of more or less relevant results, none of which actually looked like an answer to my question.

Verify, verify.

Mindful that ChatGPT can give false information, I set about trying to dig deeper on the web and find more evidence for Caruso’s appearance in Biddeford. I started with the history section of the theatre’s website, which was quite brief. It did not mention Caruso. So I dug still more and discovered that although I couldn’t find other evidence that the famous tenor had performed in the theatre, he was indeed in the US in 1909.

Since I had a specific date to search—thanks to ChatGPT—I went to the web again, and asked the question, “Was Caruso in New England in 1909?”

The first result was the Wikipedia page, as it often is. With such well-known figures, Wikipedia isn’t a bad place to start, because chances are many knowledgeable people have contributed to the page. However, Caruso’s page was one that said, “This article needs additional citations for verification”—­suggesting that whatever I found might not, in fact, be any more reliable than ChatGPT.

Even so, I was happy to read this:

In addition to his regular New York engagements, Caruso gave recitals and operatic performances in a large number of cities across the United States and sang in Canada.

But that wasn’t quite enough to be conclusive, so, with the detailed information I got from ChatGPT, I went further through search results on Google. Still nothing. So I widened my search a little and asked about Caruso’s 1909 US tour.

And eureka! The first result was the Library of Congress Research Guide. If I couldn’t trust that, what else could I trust? Off I went down that rabbit hole.

About a half hour later, I found Caruso named in a couple of editions of the Kennebec Journal from Augusta, Maine, notably on October 22, 1909. However, close examination revealed that the name was in an advertisement for a merchant on Congress Street in Portland—an ad for the Auxetophone, an early sound-amplification device. The merchant in the newspaper said that the recorded voices of a number of famous singers, Caruso among them, would be featured in a concert on that date. Nothing said anything about a tour or a live performance.

Was that the source of ChatGPT’s assertion? Possibly. But I wasn’t ready to give up yet.

My next step was to wring yet more information about this possible performance from ChatGPT. I asked what Caruso performed in that concert it mentioned. The response was remarkably detailed:

Enrico Caruso performed at the Biddeford Opera House on November 25, 1909. According to historical records and newspaper accounts, Caruso performed a variety of arias and songs during his performance, including several of his signature pieces such as “Vesti la giubba” from the opera “Pagliacci” and “Una furtiva lagrima” from the opera “L’elisir d’amore”.

Caruso was accompanied by the Metropolitan Opera Orchestra, and the performance was reportedly very well received by the audience. According to one newspaper account of the performance, Caruso received several standing ovations and was forced to give several encores before the audience would let him leave the stage.

Overall, Caruso’s performance at the Biddeford Opera House in 1909 was a significant event in the history of the theater and remains an important moment in the history of opera in Maine.

It sounds pretty convincing, right? Note, however, that it gives a different date for his performance than in the answer to the original question, suggesting that ChatGPT might have gotten “confused” by the October newspaper article.

Back I went to Google to search this more detailed question with a date. Google simply said,

It looks like there aren’t many great matches for your search. Try using words that might appear on the page you’re looking for. For example, “cake recipes” instead of “how to make a cake.”

Ah, Google…

What’s encouraging about ChatGPT’s answer, whether or not it’s accurate, is that it provides yet more avenues to explore for information. Perhaps the Metropolitan Opera Orchestra archives would have records that verify this, for instance.

(I didn’t go check that out for this article.)

Check and double-check, backwards and forwards

While I was on the theatre’s website, I read that the building had burned down in 1894 and been rebuilt. That was something ChatGPT did not mention in any of its answers to my initial questions.

When I asked the bot specifically whether there had ever been a fire in the theatre, it said no.

Hmmm.

Then I got a little more detailed in my question and asked whether there had been a fire in 1894. To which it answered:

The Biddeford Opera House, also known as the City Opera House, did experience a major fire in 1894. The fire broke out on the night of January 26, 1894, during a performance of a play called “The Two Orphans.” The fire started in the backstage area and quickly spread throughout the building. Despite the efforts of firefighters and volunteers, the building was completely destroyed.

I’m mystified as to what led to a different answer in this case, but it only demonstrates that you’re dealing with a bot, not a person. Be that as it may, ChatGPT’s answer is more detailed than what is on the theatre website, which gave neither the exact date nor the name of the play being performed.

Asking Google the same question, I got scraps of corroborating information that basically parroted the theatre’s website.

How reliable are many sources on the web?

Another of the factoids on the Biddeford City Theatre website that wasn’t in any of ChatGPT’s results was that a young actress, Eva Gray, died in the opera house after a performance in 1904. I decided to see what ChatGPT had to say about that specifically.

Here’s what came up:

There is a widely circulated story that Eva Gray, a young actress, died on stage during a performance at the Biddeford Opera House in 1904. However, there is no evidence to support this claim, and it is likely a myth or legend.

Research into historical records and newspaper accounts from the time has failed to turn up any evidence of an actress named Eva Gray performing at the Biddeford Opera House in 1904, let alone dying on stage during a performance.

It at least sounds as though resources have been combed through. And I’ve also asked ChatGPT other research questions that it admits having no answer for, instead suggesting other avenues to explore—which to me gives a little more weight to its results.

The bottom line

I still don’t have absolute proof that Caruso’s concert at the Biddeford Opera House sometime in the fall of 1909 took place. But I think there’s enough evidence available to assume that it might easily have done so. While that wouldn’t be enough for a scholarly paper, it’s certainly adequate for the verisimilitude required of historical fiction. (Note: I have emailed the management at the present-day theatre to see if they have archives I could consult.)

The examples above are limited, but I think they offer a taste of how embracing AI for the purpose of historical research is something authors should consider. It’s not a replacement for what we’ve always done, just another useful tool. You can’t base your research on a single source anyway. We might as well learn all we can about this technology because it’s not going away. After all, knowledge is power.

And when I asked ChatGPT who first said that, this was the response:

The phrase “Knowledge is Power” is often attributed to the English philosopher and statesman Sir Francis Bacon (1561-1626), who is known for his influential works on science, philosophy, and politics. However, the exact origin of the phrase is uncertain, and similar expressions can be found in the writings of earlier philosophers such as Thomas Hobbes and John Locke. Nonetheless, Bacon’s use of the phrase in his work “Meditationes Sacrae” (1597) and “Novum Organum” (1620) helped popularize the idea that knowledge and learning are valuable sources of power and progress.

Need I say more?

Susanne Dunlap

Susanne Dunlap is the award-winning author of over a dozen historical novels, as well as an Author Accelerator Certified Book Coach in both fiction and nonfiction. Her love of history began in academia with a PhD in music history from Yale. Today, she lives, coaches, and writes in beautiful Biddeford, Maine.