Evaluating the Feasibility of Automation in Library Reference Services Using LLM-Driven Text Classification

Evaluating the Feasibility of Automation in Library Reference Services Using LLM-Driven Text Classification

Academic libraries are increasingly exploring the use of artificial intelligence (AI) to streamline reference services and provide support beyond standard operating hours. This study proposes a structured framework for identifying which types of reference questions can be automatically answered by an AI-powered chatbot and validates this approach through the development and testing of a working prototype in a U.S. academic library setting. Using eight years of archived online reference transcripts from a mid-sized U.S. university library, we created a classification schema—Quick Facts, Research Help, Local Services, and Other—based on the source and complexity of each question. A fine-tuned OpenAI o3 large language model (LLM) was used to categorize each inquiry, which was then cross-referenced with the library’s Integrated Library System (ILS) APIs to assess answerability. We found that Quick Facts questions, which draw on data accessible through ILS APIs, made up 12.05% of all inquiries and could be fully answered by an automated chatbot. By incorporating a localized semantic search layer using Retrieval-Augmented Generation (RAG), the chatbot could also answer Local Services questions that require institution-specific information, raising the total proportion of answerable questions to approximately 20%. These findings suggest two key insights: first, that the effectiveness of an AI chatbot in library reference services is directly linked to the depth of its integration with ILS and local information sources; and second, that such a chatbot could reliably handle up to one-fifth of reference questions—helping to reduce repetitive workload, conserve staff time, and extend services beyond business hours. The classification framework and prototype architecture developed in this study offered a replicable model that other academic libraries can adapt to fit their own ILS infrastructure and user needs.

Qu, Meng

2025-06-29

https://creativecommons.org/licenses/by/4.0/

application/pdf

eng

Text; Poster