A Multi-Criteria Evaluation of User Feedback

• Department of General Psychology, University of Padova, Padova, Italy [elena.barzizza@unipd.it]

• Department of Management Engineering, University of Padova, Vicenza, Italy [nicolo.biasetton@unipd.it, riccardo.ceccato.1@unipd.it]

Keywords: Ranking – Multivariate – Multi-aspect

Customer satisfaction is a cornerstone of competitive advantage in today’s dynamic marketplace. Businesses recognize that positive customer experiences drive profitability [Singh, 2006] and solidify market position [Aaker and Moorman, 2023]. The subjective and multifaceted nature of satisfaction makes it difficult to analyze. While surveys employing Key Performance Indicators (KPIs) offer valuable insights into customer sentiment, effectively aggregating and interpreting this data to generate meaningful product or service rankings presents a significant challenge. This paper introduces a novel multi-aspect ranking methodology designed to address this challenge by incorporating both the average level of satisfaction and the consistency of customer evaluations across multiple KPIs.

Traditional rankings often rely solely on average scores, ignoring how consistent customer ratings are. A product with consistently good ratings is often superior to one with a higher average but more varied feedback. Our proposed approach addresses this limitation by integrating a multi-aspect permutation test based on the NonParametric Combination (NPC) methodology [Pesarin and Salmaso, 2010] with an existing ranking technique [Arboretti et al., 2014]. This innovative combination allows for a more nuanced and robust assessment of customer satisfaction by simultaneously considering multiple KPIs and various aspects of statistical testing, including both measures of central tendency and variability. This method is able to discern not only which product or service achieves the highest average satisfaction scores, but also which demonstrates the greatest consistency in positive customer feedback.

To demonstrate the practical applicability and effectiveness of our proposed methodology, we present a real-world case study involving customer reviews for three different streaming video services (Service A, Service B and Service C). User feedback was collected across three key performance indicators: content library, user interface, and value. These KPIs were chosen to provide a holistic assessment of the user experience, encompassing the breadth and quality of available content, the ease of navigation and use of the platform, as well as the perceived value for money. Our analysis aimed to identify the streaming service that not only achieved the highest average ratings but also demonstrated the most consistent user satisfaction across these key dimensions.

A close look at the data reveals that Service A boasts a greater proportion of top-tier ratings across all three key areas, suggesting a generally more favorable user sentiment. Services B and C, on the other hand, display a more dispersed pattern of ratings, encompassing a broader spectrum of scores. This is confirmed by the multi-aspect based ranking procedure which gives back the following ranking (from the best to the worst): Service A, Service B and Service C. This case study demonstrated the practical value of our approach in a real-world setting, showcasing its ability to provide accurate and insightful product rankings.

In conclusion, this paper introduces a novel multi-aspect ranking procedure that offers a significant advancement in the assessment of customer satisfaction. By integrating the NonParametric Combination methodology with an established ranking framework, our approach provides a more comprehensive and robust evaluation of product or service performance. The incorporation of both central tendency and variability measures allows a deeper understanding of customer sentiment. The real-world case study demonstrates the practical applicability of our method and highlights its potential to provide valuable insights for businesses seeking to enhance customer satisfaction and gain a competitive edge. A simulation study is ongoing to further validate the performance of the proposed methodology.