Enhancing News Retrieval with Diversity and Interpretability

Abstract:

The rapid shift toward digital news consumption has made information retrieval systems central to how users access, interpret, and engage with current events. While these systems enable scalable access to vast news corpora, they also raise critical concerns about transparency, redundancy, political bias, and user trust. In particular, conventional retrieval pipelines—often optimized solely for relevance—frequently return results that are redundant, lack viewpoint diversity, and offer limited interpretability into how rankings are determined. These limitations are especially consequential in high-stakes domains such as news, where balanced perspectives and transparent reasoning are essential for informed public discourse.

This thesis addresses these challenges by proposing a suite of methods that enhance two under-explored properties of news retrieval systems: diversity and interpretability. Focusing on both stages of the retrieval pipeline—representation learning and search—we introduce three core contributions:

First, we propose DkMIPS, a general framework for Diversity-aware k-Maximum Inner Product Search, which balances relevance and inter-document diversity. This method improves the quality of ranked results by minimizing content redundancy and surfacing complementary perspectives across retrieved articles.

Second, we develop CQG-MBQA, a novel framework for generating interpretable semantic text embeddings using contrastive question generation and multi-task binary question answering. This approach enables embedding-based retrieval to produce not only semantically relevant results but also human-understandable explanations of document-query alignment.

Third, we introduce PRISM, a method for producing interpretable political bias embeddings that encode ideological dimensions of news articles using politically aware cross-encoders. PRISM supports bias-aware retrieval and downstream analysis of ideological bias.
These methods are integrated into a prototype system, DiversiNews++, which combines diversity-aware search (DkMIPS), interpretable embedding-based interpretation (CQG-MBQA), and political bias modeling (PRISM). The system provides interactive visualizations to help users understand the relevance, diversity, and coverage rationale behind retrieved articles, offering a transparent and comprehensive news reading experience.

Together, these contributions advance the goals of diverse and interpretable news retrieval, addressing the growing need for trustworthy information access in an era shaped by algorithmic content delivery.