Structured Multimodal Intelligence: Scene Graphs for Vision-Language Understanding, Generation, and Unification
COM1 Level 3
MR1, COM1-03-19
Abstract:
To achieve the ultimate goal of human-level artificial intelligence (AI), a fundamental prerequisite is the ability to understand and generate diverse forms of information in a naturally interactive and grounded manner. Among all modalities, vision and language stand as the most fundamental forms that encapsulate human experiences, making vision–language (V–L) learning a cornerstone for developing machines capable of perceiving, reasoning, and interacting in real-world environments. While recent advances have yielded powerful V–L systems, significant challenges remain in achieving the fine-grained understanding, controllable generation, and unified modeling necessary for scalable and interpretable multimodal intelligence.
A major obstacle lies in the representational gap and modeling paradigm. On the one hand, visual representations are inherently dense and spatial, whereas linguistic representations are abstract and sequential, creating a structural and semantic mismatch that complicates cross-modal learning. This obstacle is further compounded by the prevalent use of global-level alignment, which treats images and text as holistic entities. Such a design results in suboptimal fine-grained grounding, restricted controllability in generation, and weak coupling between perception and generation. Specifically, this thesis focuses on three key research challenges: (1) cross-modal fine-grained misalignment, where models fail to ground visual and linguistic elements at the object and relational level; (2) lack of controllability in generation, leading to coarse, semantically inconsistent, or hallucinated outputs; and (3) absence of a modality-invariant structured representation, hinders the seamless V-L integration of understanding and generation within a shared framework.
To tackle these challenges, we propose a scene graph-empowered modeling paradigm that positions structured semantic representations as the core abstraction for V–L learning. Based on this, we detail the proposed solutions.
Firstly, for fine-grained vision–language understanding, existing methods often remain confined to global-level alignment or limited object-level matching, while overlooking the semantic-level asymmetry between modalities during information fusion. We address this by representing the semantic structures of both visual and textual inputs as scene graphs, enabling alignment at the object, relation, and temporal levels. This framework is validated on both text–image and text–video tasks, including fine-grained multimodal relation extraction and video question answering, consistently outperforming strong baselines.
Secondly, a major challenge in cross-modal generation lies in the lack of fine-grained controllability, stemming from the inherent asymmetry between abstract, compact language descriptions and dense, detail-rich visual content. This mismatch often leads to attribute misbinding, hallucinated objects, and violations of spatial or relational constraints implied by the prompt. We address this by introducing a scene graph–driven framework that serves as a structured, interpretable, and modality-bridging scaffold, explicitly encoding objects, attributes, relationships, and temporal dynamics to guide the generation process. By expanding sparse textual prompts into semantically rich and coherent scene layouts and integrating them into diffusion-based generators, our approach enables precise control over spatial configurations and semantic content. Experiments demonstrate substantial gains in semantic fidelity, spatial consistency, and controllability over strong baselines in both static and dynamic generation settings.
Thirdly, beyond advancing understanding and generation separately, achieving unified vision–language modeling in both task paradigm and modality space is essential. A major obstacle lies in the lack of a modality-invariant structured representation capable of bridging the modality, structural, and semantic gaps between vision and language. We introduce the Universal Scene Graph (USG), a modality-agnostic, compositional abstraction that captures shared semantics while preserving modality-specific nuances. Serving as an interpretable and grounded intermediate layer, the USG enables unified encoding, reasoning, and generation within a USG-empowered Multimodal Large Language Model (USG-MLLM), treating vision and language as first-class citizens in both understanding and generation. Experiments demonstrate that this framework significantly improves cross-modal generalization, semantic consistency, and reasoning flexibility over state-of-the-art unified V–L models.
To sum up, extensive experiments across benchmark datasets validate the proposed methods' effectiveness in improving interpretability, controllability, and generalization for vision-language learning. This thesis demonstrates that structured scene graph representations offer a scalable and principled pathway toward general-purpose, unified vision–language foundation models.