Performance Analysis of Fine-Tuned ChatGPT-3.5 Chatbot for Alni Accessories E-Commerce Service

Abstract

Background: The rapid advancement of artificial intelligence (AI) technology, particularly in language modeling, has driven the adoption of chatbots to support e-commerce services. Alni Accessories store faces challenges related to limited product information and slow customer service, which negatively impact user satisfaction.
Aims: This study aims to analyze the performance of a fine-tuned ChatGPT-3.5-based chatbot to improve interaction quality and transaction efficiency on the e-commerce platform.
Methods & Results: The evaluation process was carried out through a series of experiments, where the 13th experiment achieved the best results with a Training Loss of 0.3894 and a Validation Loss of 0.5787. The Training Mean Token Accuracy of 0.8799 and Validation Mean Token Accuracy of 0.7971 indicate the model’s ability to effectively learn language patterns. Furthermore, the Full Validation Loss of 0.6242 and Full Validation Mean Token Accuracy of 0.8122 demonstrate the model’s strong generalization capability. The independent t-test produced a p-value of 0.0001, confirming a statistically significant difference in transaction response time between the old system and the new system. The findings of this study show that the fine-tuned ChatGPT-3.5 chatbot not only accelerates services and improves product information accuracy but also holds great potential for implementation in other sectors such as education and healthcare.