Advancing Language Learning Systems from Acoustic to Audio-Language Modelling
AS6 Level 5
MR6, AS6-05-10
Abstract:
Learning a second language is a common need within bilingual or multilingual communities. Traditional computer-assisted language learning (CALL) and computer-assisted pronunciation training (CAPT) systems offer valuable support; however, they rely on low-level acoustic features, which often lack robustness, and provide minimal feedback beyond numerical assessment metrics. This thesis investigates the integration of large-scale pre-trained acoustic, language, and audio-language models to advance the development of modern language learning systems.
Firstly, the thesis explores the application of large pre-trained acoustic models for CAPT, specifically utilizing models such as HuBERT for pronunciation assessment. We introduce a novel zero-shot automatic pronunciation assessment method, leveraging the consistent features of self-supervised models trained on masked token prediction tasks. To address the challenges posed by acoustic distribution shift, we further propose a test-time adaptation method, enabling the pre-trained acoustic models to adapt under wild acoustic test settings including speaker variability, additive environmental noises, and pronunciation variations such as accents and sung speech.
Secondly, this thesis examines the incorporation of large language models (LLMs) for CALL. To enhance the in-context learning of LLMs for downstream tasks in CALL, we propose to construct informative few-shot prompts and select prompts with maximum information gain. Additionally, we develop a human-AI cooperative framework, ChatCoach to assist language acquisition for early-stage medical learners and early-career practitioners. ChatCoach allows the detection and correction of language misuse in medical domains during conversations.
Lastly, this thesis investigates the development of audio-language models for both CAPT and CALL. We propose joint training of pre-trained audio encoders with large language models by modality alignment and task-specific instruction tuning. This approach enhances the models with joint acoustic understanding and mispronunciation detection, offering learners more comprehensive feedback with detailed explanations and actionable suggestions, extending beyond simple numerical evaluations.