My Discussion with ChatGPT

I had the opportunity to check out ChatGPT this week. Honestly, I didn’t have high expectations. There has been so much chatter about this tool, that I was sure that there was more hype than substance. I decided to give it an hour to see what it was all about.

My very first impression was that it is actually much more like a research project or proof of concept than a product. I started early in the day and received a message indicating that they have gotten a lot of interest and are in the process of scaling up their servers. One nice bit of design is that this message didn’t just appear, it displayed word by word, as if someone was thoughtfully typing me a response. I found this style to slow things down a bit. I actually read every word rather than my regular scan to get the just before moving along. I later found that this is part of the style of ChatGPT - it types out the responses for you letter by letter, and the result is that using it feels more like a long-form conversation on Slack than an interaction with, what I’ll call, a next-generation search engine. The style does have you reading the answers as they are being displayed, and gave me the sensation that I was having a conversation rather than searching for data.

After several hours, I was finally able to create an account and ask my first question. I’ve been doing research for a keynote talk that I’ll be giving in June, which is about how digital natives (who age-wise are just about PhD-earning age) might disrupt traditional methods for scholarship, research, and knowledge creation. This topic seemed like a perfectly ironic one for doing some research about an AI chatbot.

Research by search engine vs ChatGPT

My normal process for researching for a talk or paper is to develop strategic queries for my searches to fill in gaps in my knowledge or support theories that I have. I scan the list of websites and papers presented, open a few to assess relevancy and where the site’s information fits on the research/fact-to-opinion scale. If I don’t find anything useful on the first two pages of search results, I try again with different search terms. Once I have a few sites with info, I read the materials in more depth and summarize in my notes what I learned, carefully including the source of the information so that I can refer back to it later. Some information I ultimately use, and other I file away in my personal knowledge bank.

I was curious how this process might change with ChatGPT. Immediately apparent was that I didn’t receive a suggested set of websites that might answer my questions, instead, my queries resulted in the summarized notes that I may have jotted down after my research process was complete. This led me to probe further on this summarized result with further questions to deepen the query. Between the conversational style of ChatGPT and this rapid-fire ask a question - receive what seems to be a neatly-formed, researched answer, the experience felt more like talking with someone at a cocktail party who had some knowledge about the topic. I got succinct, summarized answers to my questions and the opportunity to ask follow up questions. But, also like a cocktail conversation, I didn’t get any hard fact, citations of studies or research to support what was said, or any real way to to assess what was “true” - it just was what this seeming expert told me when I asked the question.

But, you know, sometimes that type of interaction is just perfect. In this case, it gave me the opportunity to test the boundaries of of my thesis, consider a few points that I wasn’t thinking about, and helped organize my partially-formed thoughts into the neat, number lists that are the generic style of ChatGPT. As it stands today, this tool can’t replace my research methods - there is not enough evidence or detail for me to pick through it and come to conclusions myself. But, I think it is a very useful tool when in the thesis forming stage in exploring what “conventional wisdom” might be or sort out the areas where I might be skeptical.

Would I try it again?

I definitely would try it again. But, I may have to wait until they upgrade those servers. Getting a “network error” in the middle of a chat feels a bit like those old-school TV series cliffhangers of yesteryear. That just won’t do when I’m trying to get a talk or paper out the door!