Generative AI at Work
Insights from a large-scale field experiment on customer-service agents
A study by Brynjolfsson, Li, and Raymond titled “Generative AI at Work,” published in The Quarterly Journal of Economics, provides a rich case study that illustrates this point. The authors introduce the question of implementing AI technologies as follows:
“The emergence of generative artificial intelligence (AI) has attracted significant attention for its potential economic impact. Although various generative AI tools have performed well in laboratory settings, questions remain about their effectiveness in the real world, where they may encounter unfamiliar problems, face organizational resistance, or provide misleading information (Peng et al. 2023a; Roose 2023).”
The Study in Brief
They examined the implementation of a generative AI conversational assistant among 5,172 customer support agents “working for a Fortune 500 firm that sells business-process software.” Regarding the focus of their study, they write:
“We study the effect of generative AI on productivity and worker experience in the customer-service sector, an industry with one of the highest surveyed rates of AI adoption (…). The tool we study is built on Generative Pre-trained Transformer 3 (GPT-3), a member of the GPT family of large language models developed by OpenAI (OpenAI 2023). The AI system monitors customer chats and provides agents with real-time suggestions for how to respond. The AI system is designed to augment agents, who remain responsible for the conversation and are free to ignore or edit the AI’s suggestions.”
They found that access to the AI tool increased productivity, defined as issues resolved per hour, by approximately 15% on average. Importantly, the gains are not uniform: less experienced and lower-skilled agents show the largest relative improvements, whereas the most experienced / high-skilled agents show little change in productivity and small declines in quality. They explain:
“The gains accrue disproportionately to less experienced and lower-skill customer-support workers indicating that generative AI systems may be capable of capturing and disseminating the behaviors of the most productive agents.”