Historically, I’ve been a late adopter of technology. I was one of the last people I know to get a cell phone. And I continued to pay for gas inside the gas station for years after everyone else started paying at the pump with a credit card. But recognizing my own Luddite tendencies, I try to deliberately challenge my biases every once in a while.
ChatGPT, the newest general-purpose chatbot, seemed like a good test.
I was skeptical of ChatGPT before I even knew it existed. Like you, I have encountered numerous special-purpose chatbots on various websites. In general, I found engaging with these chatbots unhelpful, annoying, and time-wasting. No matter how trivial my request, the usually unhelpful response left me with the distinct impression that I was either crazy or being gaslit.
My skepticism was reinforced by some early reviews, such as that of Paul T. von Hippel, who wrote,
ChatGPT is often wrong but never in doubt. It acts like an expert, and sometimes it can provide a convincing impersonation of one. But often it is a kind of b.s. artist, mixing truth, error, and fabrication in a way that can sound convincing unless you have some expertise yourself.
But other users are much more optimistic. Writing here on Minding the Campus, Jonathan Sircy made a compelling case that ChatGPT can have many useful educational applications:
ChatGPT provides a solution to this problem by giving students personalized assessments that will improve their reading, writing, and thinking skills. The tool offers the educational freedom of bespoke feedback at scale. …
If the student provides a prompt, “Generate an exercise for me that would help me improve my use of transitional words,” the bot does just that. If the student asks the bot to provide feedback on the completed exercise, it will. …
[R]eal-time responses are an essential component of deliberate practice, the method of mastering any skill. ChatGPT offers a scalable method of reading, writing, and thinking instruction that can supplement traditional methods, making it a valuable resource for students and teachers alike. Yes, we know how to abuse the tool, but we should acknowledge its potential too.
Others such as Tyler Cowen and Arnold Kling are extremely bullish on this new technology. While these endorsements remind me a little bit of this famous scene, they did convince me to try ChatGPT for myself.
I have a few projects in various stages of progress, so I engaged with ChatGPT on several of them. Here are the four main lessons I learned from using ChatGPT—two don’ts and two dos.
1. Don’t ask ChatGPT to settle factual disputes.
It hedges quite a bit, it’s easily swayed by the way you ask the question (much like humans), and it will fabricate an answer if one isn’t readily available. None of these responses make it very useful as an arbiter of truth, or even of conventional wisdom.
2. Don’t ask ChatGPT for reading suggestions.
I asked for some journal article and book recommendations on a topic that I knew well, and the results were barely relevant. Interestingly, ChatGPT will occasionally make up fictious works, such as when answering “What is the most cited economics article?”
3. Do ask ChatGPT to summarize the pros and cons of arguments.
I have a forthcoming paper comparing state appropriations with student aid, so I asked the bot: