Document-Level Relation Extraction and Temporal Reasoning with Large Language Models
COM3 Level 2
SR21, COM3 02-60
Abstract:
Large Language Models (LLMs) have become the backbone models for many natural language processing (NLP) tasks. In this thesis, we study the applications and limitations of LLMs in two aspects: (1) Document-level Relation Extraction (DocRE), and (2) Temporal Reasoning.
LLMs have drastically improved the performance of the relation extraction task. However, document-level relation extraction is still a very challenging task even when LLMs are adopted. Different from conventional sentence-level relation extraction, document-level extraction aims to extract the relations among all entities within one document. Since the complexity of the relation extraction task is quadratic in the number of entities, document-level RE is far more challenging than its sentence-level counterpart, as the average number of entities in a document is significantly higher than that in a typical sentence. The state-of-the-art performance of the DocRED leaderboard was less than 66 F1 before November 2019. we introduce three of our works in DocRE. We first introduce our novel training framework for DocRE, which achieved state-of-the-art performance on the DocRED leaderboard. Secondly, we conducted an in-depth analysis of the most popular DocRE dataset and identified the false negative problem within the dataset. Our analysis showed that approximately two thirds of the relations are not annotated in the original dataset. We then proposed a machine-assisted annotation pipeline to re-annotate the DocRED dataset and provided a cleaner version of the original dataset. Experimental results showed that a model trained on our dataset is able to achieve around 13 F1 performance increase. Thirdly, we study the problem of DocRE when the training data is incompletely annotated. We proposed the class-adaptive self-training (CAST) algorithm to re-sample the pseudo-labels based on the per-class performance of RE. In this way, we are able to improve the overall recall without significantly compromising precision.
Next, we study the temporal reasoning aspect of LLMs. We first provided a detailed review on the related work. We exposed the biases in LLMs' temporal reasoning in favor of the contemporary years. We then proposed the \tempreason dataset, a comprehensive evaluation benchmark that contains three levels of temporal reasoning: (1) Time-time reasoning, (2) Time-event reasoning, and (3) Event-event reasoning. We also proposed a training framework to improve the temporal reasoning capability of LLMs. In the third section, we study the problem of multi-hop temporal reasoning and the temporal co-occurrence phenomenon, which are the under-explored aspects in temporal NLP research. We proposed a new dataset Complex-TR with more comprehensive evaluation metrics to investigate complex temporal reasoning. We showed that LLMs are not able to perform complex temporal reasoning well and the reasoning capability is not robust to temporal shifts. To address these limitations, we proposed Pseudo Instruction-tuning (PIT), a novel data augmentation strategy to generate temporal reasoning data. Our PIT approach is able to significantly improve the multi-hop temporal reasoning performance, especially for the low-resource years.
In addition, we outline the proposed work to be carried out before thesis submission and provide a timeline for its completion. The proposed work involves improving the retrieval performance of open-domain temporal question answering and utilizing tool-assisted LLMs.