Grounding LLM Evaluations in Social and Cultural Context
COM3 Level 2
SR21, COM3 02-60
Note: Please register for the event here: https://forms.office.com/r/3ZysJjF87z
Abstract:
Evaluating LLMs for subjective tasks such as coaching, therapy, and advice giving requires knowing what a "good" response looks like, which depends on cultural and social context. We first demonstrate an interpretable framework for comparing the linguistic cues that LLMs and humans use to signal politeness across languages. We then examine style preservation in cross-cultural translation, finding that LLMs perform worst in non-Western languages, where they systematically bias translations toward stylistic neutrality, a failure invisible to standard metrics. We address this with RASTA, a retrieval-based method that improves culturally appropriate style preservation without degrading content quality. Finally, we introduce the Culturally-Aware Conversations benchmark. Overall, current frontier LLMs still struggle to converse appropriately across non-Western cultures.
This is a joint work with Shreya Havaldar.
Bio:
Lyle Ungar is a Professor of Computer and Information Science at the University of Pennsylvania, where he also holds secondary appointments in Psychology, Bioengineering, Genomics and Computational Biology, and Operations, Information and Decisions. His group develops natural language processing and explainable AI for psychological and medical research, including analyzing social media to better understand the drivers of physical and mental well-being and building socio-emotionally sensitive AI-based tutors and coaches.