Enhancing Conversational Agents: Source-Aware Generation and Conversational Partner Understanding
COM2 Level 2
MR3, COM2-02-26
Abstract:
Effective dialogs pivot on the conversational agent’s abilities in integrating diverse information and knowledge sources and understanding the perspectives of the conversational partners to craft responses that are well-informed and coherent. This proposal delves into enhancing conversational agents through source-aware generation and conversational partner understanding.
For source-aware generation, we explore two types of information sources—text documents and images/videos, in which topics of dialogue-aware knowledge retrieval and modality alignment are investigated. For text document sources, we present two approaches to retrieving knowledge from documents for dialogue response generation. First, we employ knowledge sequence modeling to ensure both knowledge coherence and user engagement in the individual responses generated. Then, we exploit the structured knowledge from documents to seamlessly integrate dialogue contexts and knowledge structures to guide the overall conversational flow effectively. Moving on from single-modal text documents to multi-modal images and videos, we introduce a work that aims to seamlessly connect the video modality with language-based models through a parameter-efficient technique, which largely improves video incorporation by introducing a suitable amount of pooled video features, and resists the degradation of language ability during training by fused LLM weights.
For conversational partner understanding, we consider two dialog settings: one on task-oriented dialogs and another on multi-agent strategic game-playing settings. For task-oriented dialogs, we aim to equip the conversational agent with a robust memory of the dialogue status, enhancing its ability to make informed decisions with a better understanding of the user’s intent. For the more complex multi-agent dialog setting, we devise a probabilistic graphical model-based mutual dialogue memory to capture the agent’s understanding of multiple conversational partners to enhance its conversational ability to achieve complex goals such as fostering collaboration for competition within multi-agent systems.
Overall, this thesis investigates multiple key aspects in enhancing conversational agents through source-aware generation from document to image/video response generation and conversational partner understanding from dialog state tracking for task-oriented dialogs to collaborative learning within a network of multiple agents.