Temporal Action Segmentation in Complex Activities Videos
Abstract:
Temporal Action Segmentation (TAS) involves temporally localizing and classifying actions within complex activity videos. It assigns action labels to each frame, ensuring accurate recognition of actions as they occur. Unlike action recognition in trimmed videos, which focuses on identifying a single action, TAS handles untrimmed videos containing multiple actions occurring in sequence. Examples include “making coffee”, “making eggs”, “performing CPR”, or “changing a tire”.
Temporal convolutional networks (TCNs) are widely used for TAS but often suffer over-segmentation errors, requiring additional refinement for smoothness and temporal consistency. To address this, we propose an encoder–decoder TCN called C2F TCN, featuring a “coarse to fine” ensemble of decoder outputs and a model agnostic temporal feature augmentation strategy using stochastic max pooling of segments. This yields more accurate and smooth results across three benchmark datasets.
Given the high cost of fully supervised learning, which requires dense frame wise labeling, we introduce the first semi supervised method for TAS. Our approach needs dense annotation only for a few labeled videos. It leverages unsupervised contrastive representation learning from TCNs, using input feature clusters and multi resolution similarity. This is integrated into an “Iterative Contrast Classify (ICC)” semi supervised learning scheme. ICC progressively improves with more labeled data; using 40% labeled videos, its performance approaches that of fully supervised models.
To further reduce annotation density within videos, we develop a sparsely labeled learning method based on expectation maximization. This learns from videos thousands of frames long but with only a few labeled frames (as few as 7 to 32). This approach requires some labeled frames in every video and thus is a form of weak supervision. In such a setting, reducing the number of videos undercuts performance. So, we devise an algorithm that combines semi supervised and weakly supervised settings, labeling only a few selected frames from a few videos. Under traditional semi and weak supervision, videos and frames are selected at random. With smaller annotation budgets, it becomes more advantageous to actively select which videos and which frames to label. This brings us to our work on active learning for TAS. We design Coherence Enforcing Active Learning (CEAL), which selects key frames and videos based on segment coherence and uses generalized EM to infer missing labels. CEAL exceeds other SOTA weak, semi and active learning setups when annotation budgets are comparable and achieves performance comparable to fully supervised models with under 1% of frames labeled.
In conclusion, our supervised model and semi , weak , and active learning strategies significantly reduce annotation burden while achieving competitive performance in temporal action segmentation of complex videos.