Temporal Modeling for Procedural Video Understanding
COM1 Level 2
SR7, COM1-02-07
Abstract:
Procedural video understanding involves analyzing long, untrimmed videos containing multiple actions, where modeling temporal dependencies is crucial for accurate detection and interpretation. However, challenges such as action imbalance and incomplete contextual information hinder effective temporal modeling. This thesis addresses these issues in two settings: temporal action segmentation under imbalanced action distributions, and online action detection with incomplete knowledge, where future frames are unavailable.
The study first focuses on temporal action segmentation, which densely labels frames in untrimmed procedural videos. These videos often exhibit long-tailed action distributions, with actions varying significantly in frequency and duration. Such imbalance introduces learning biases that hinder the modeling of infrequent action transitions. As a remedy, we propose a constrained optimization approach with a cost-sensitive loss function. This function dynamically adjusts weights based on the learning state of actions and transitions, ensuring a more balanced modeling of actions and transitions.
We further observe that existing long-tail methods tend to break temporal dependencies due to class-independent assumptions. To address this, we propose addressing the long-tail issue by leveraging the temporal dependencies themselves. Specifically, we introduce a novel group-wise temporal logit adjustment framework that combines a group-wise softmax formulation based on high-level task information with temporal logit adjustments based on action ordering. By leveraging action inter-dependencies, the proposed framework improves in segmenting tail actions without compromising performance on head actions, preserving the learned temporal dependencies.
Next, we explore temporal modeling in online action detection, where actions are identified frame by frame in streaming videos using only past observations. While current approaches usually focus on capturing temporal dependencies from past data and rely on anticipation to handle the lack of future information, we highlight the importance of modeling immediate temporal dependencies and introduce a context-enhanced memory-refined transformer. The proposed model enhances temporal understanding by integrating both near-past context and near-future generation, allowing for a more effective use of available information from the past, present, and anticipated future.