The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?
Blondeel, E., and Everaert, P. in Wood et al. (2023). The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions? Issues in Accounting Education, https://doi.org/10.2308/ISSUES-2023-013
Open AI, an artificial intelligence (AI) research and development company, released the beta version of ChatGPT for public use on November 30, 2022. ChatGPT is a large language-learning model that utilizes machine learning to generate natural language text. ChatGPT has garnered considerable attention for its ability to respond to users’ questions. This study provides data on how well ChatGPT performs in answering assessment questions primarily from class exams and quizzes. ChatGPT’s performance on accounting assessments is compared to student averages on the same set of questions, analyzing where it performs better and worse. A crowd-sourcing technique for data collection and paper authorship is used. By this, data from 14 countries and 186 institutions is collected.
The results of the 28,085 analyzed questions show, as of January 2023, that ChatGPT provides correct answers for 56.5 percent of questions and partially correct answers for an additional 9.4 percent of questions. When considering point values for questions, students significantly outperform ChatGPT with a 76.7 percent average on assessments compared to 47.5 percent for ChatGPT if no partial credit is awarded and 56.5 percent if partial credit is awarded. Still, ChatGPT performs better than the student average for 15.8 percent of assessments when we include partial credit. Evidence is provided of how ChatGPT performs on different question types, accounting topics, class levels, open/closed assessments, and test bank questions. The results will help academics, administrators, faculty, instructors, and students better understand the potential impact of large language-learning models like ChatGPT on education, not only in the field of accounting, but also in many fields with similar types of assessments such as finance, information systems, management, and operations.
Read more here.