Cross-Modal Priors for 3D and Event-Based Vision: Repurposing Foundation Models in Data-Scarce Regimes
Abstract:
2D vision has advanced rapidly due to the availability of large annotated datasets that support supervised learning at scale. Alongside this dominant paradigm, alternative visual modalities such as event based sensing and neural scene representations offer capabilities that exceed those of conventional 2D images. Event based perception captures fine-grained motion dynamics and remains reliable under extreme illumination, while neural scene representations model the geometry and appearance of the physical three dimensional world with high fidelity. These modalities excel in domains where conventional vision systems fail. Event cameras enable robust operation under extreme conditions, supporting applications in gesture recognition, drone navigation, and biomedical imaging. Neural scene representations are central to immersive AR/VR, photorealistic rendering, and digital twin simulation. Yet progress in these modalities is often constrained by the scarcity of high quality, modality specific annotations needed to fully exploit their strengths.
This thesis proposes leveraging visual priors from foundation models to compensate for the data gap in these alternative visual modalities. To this end, we introduce three frameworks that demonstrate the efficacy of cross-modal prior transfer.
The first, DiSR-NeRF, addresses the problem of super-resolving neural radiance fields from low-resolution multi-view inputs without access to any high-resolution views. It distills high-frequency details from a pretrained diffusion-based image super-resolution model using a novel Renoised Score Distillation (RSD) objective, and enforces multi-view consistency through an Iterative 3D Synchronization (I3DS) scheme.
The second, EventSR-Zero, enables event-based video super-resolution in a zero-shot setting by conditioning a frozen video diffusion model with high resolution motion cues extracted from low-resolution event data. These cues are recovered via an Implicit Contrast Refinement (ICR) module and injected into the generative process using Reconditioning Guidance (RG), enabling coherent and temporally aligned frame synthesis without task-specific fine-tuning.
The third, Distil-E2D, tackles monocular depth estimation from events by distilling dense pseudolabels from RGB-based foundation models. These are reconciled with sparse LiDAR using a Confidence-Guided Calibrated Depth Loss (CCDL) that incorporates a Nonlinear Depth Calibration module and an Alignment-aware Confidence Estimator. The model further employs a Context Transformer for temporal aggregation and a Dual-Decoder Training strategy to enhance representation learning.
Together, these frameworks introduce three strategies that establish cross-modal prior transfer as a powerful paradigm for overcoming supervision bottlenecks in emerging visual modalities, achieving state-of-the-art performance under data-scarce regimes.