OpenAI’s chatbot, ChatGPT, debuted in November and has quickly become a viral sensation. Its capabilities seem immense. From producing different styles of writing based on user prompts to writing scientific papers or even passing the exams required to practice medicine, this tool has opened a whole new world of possibilities. And with it, new fears. From venture capitalists to our friends at the bar, people seem to think AI is about to put knowledge writers and communicators, like us, out of a job. But is it, though?
To get a sense of what these tools have to offer, our team has decided to research the ones that are currently available, their aims and capabilities, and to discuss our results as we go. Our goal is to establish which ones are worth a more thorough check and find out how we can use AI to improve the quality of our work. Naturally, given its popularity, we started by investigating the performance of ChatGPT-3.
First, we decided to ask the AI for help writing a blog post, a ubiquitous format that we know inside out. The results were predictable but also surprising. As expected, the delivery was robotic and plain. The texts sounded unnatural, lacking the imagination and emotional undertone that define creative writing and attract readers. But even if it failed to produce engaging copy, some details caught our attention.
Despite the well-known criticisms, ChatGPT cited sources correctly, giving us links to the published papers when requested. It organized complex information hierarchically, pointing out the connections that articulate our understanding of these topics. And it even suggested ways of exploring ideas we hadn’t considered before. It also lied.
When we asked it to provide names of relevant researchers for a given topic, at first, ChatGPT delivered some useful results, but as we tried to refine the search it started lying. It gave us lists of fake names, together with fake titles from some very real universities. Once we pointed out the blatant lies, the AI apologised, but it failed to change its ways. It kept lying. And we are not the only ones witnessing this behaviour.
It felt like working hand in hand with an unbiased, amoral entity, that came with no regrets and some useful features to unlock our minds. It has the potential to write text with the sole purpose of helping us beat writer’s block, it can come up with ideas for a brainstorming session quickly and efficiently. That its writing abilities are limited is beside the point. The crux of the matter is that we can learn how to use this technology to our advantage.
Will ChatGPT revolutionise science communication? We certainly hope so! Just not in the way most people think. Just like we need criteria and knowledge to discern the quality of information we find online through traditional search engines, we’ll also need to evaluate anything produced by an AI model with a critical and trained eye. Yet, it may help us free up important chunks of time, allowing us to indulge the pickiness that lives inside every writer. The AI may give us more time to polish our texts, evaluating every word and every detail, and in the end, producing better writing.
