Scalable Solutions for Multivariate Time Series Forecasting

Abstract:

Multivariate Time Series (MTS) data, characterized by temporal dependencies and inter-variable correlations, encapsulate essential dynamics across diverse systems. At the heart of utilizing MTS is MTS forecasting, a powerful approach for extracting valuable insights and guiding decision-making across domains such as finance, healthcare, energy, and urban computing. Generally, MTS can be categorized into two types: Normal Time Series (NTS) and Spatial Time Series (STS). This distinction is based on whether the inter-variable correlations can be predefined using rules or prior knowledge. Due to the differences in these data characteristics, the modeling techniques for forecasting each type of MTS vary significantly.

This thesis centers on developing scalable solutions for practical use of MTS in real-world systems. The term scalable encompasses three aspects: (1) the capacity of the model to scale to large MTS data, (2) the ability of the model to manage in-distribution NTS and STS forecasting tasks using a unified MTS model, and (3) the ability of the model to handle unlimited zero-shot forecasting tasks as an MTS foundation model. In the following, we delineate the challenges in each aspect and propose innovative techniques to resolve them respectively.

In the first aspect, given that large-scale MTS data (e.g., transportation time series) often fall under the STS category, we focus on improving the scalability capacity of Spatio-Temporal Graph Neural Networks (STGNNs), which are the most popular tools for STS forecasting. The main issues stem from their reliance on Graph Neural Networks (GNNs), which introduce efficiency and scalability challenges, making them less suitable for latency-sensitive, large-scale forecasting tasks. To address these limitations, we first revisit how forecasting methods modeled variable correlations before the rise of GNNs. After studying the relevant literature, we observe that a class of methods we call node-centric methods has unique advantages in terms of fast inference time and low GPU memory cost. However, they require lengthy preprocessing time and suffer from inferior model performance. In light of this, we introduce the SimST framework, addressing their limitations while preserving or even enhancing their strengths. Extensive experiments show that SimST achieves performance on par with STGNNs while exhibiting up to 39 times inference speedup.

Then, we discuss the possibilities and the advantages of learning a unified MTS forecasting model for both NTS and STS data. Previously, the MTS forecasting literature shows that existing approaches focus on developing specialized models for different type of MTS data and for different application domains, creating a huge obstacle to deploying numerous MTS models in real-world applications. Consequently, there is a pressing need to establish a unified modeling paradigm that transcends data type and domain boundaries. To this end, we introduce UniTime, an innovative model to unify in-distribution MTS forecasting tasks. Our extensive experiments demonstrate the effectiveness of UniTime in advancing state-of-the-art forecasting performance.

Finally, we delve into the development of MTS forecasting foundation models, which are capable of tackling zero-shot forecasting tasks at an industrial scale. A key open challenge in developing such models lies in achieving effectively unified training on large-scale time series corpora. Existing approaches attempt to introduce model specialization to handle the highly heterogeneous nature of MTS data. However, their specialization strategies are often either based on human-defined heuristics or too coarse-grained to capture the diversity of MTS. To address these limitations, we propose Moirai-MoE, which leverages sparse mixture of experts (MoE) within Transformers to model diverse MTS patterns. Consequently, Moirai-MoE eliminates reliance on human-defined heuristics and enables automatic token-level specialization. Extensive experiments demonstrate the superiority of Moirai-MoE over existing MTS foundation models in both in-distribution and zero-shot scenarios.