Vision and Language Learning with Limited Resources

Abstract:

Vision-and-language (VL) understanding aims to equip machines with human-like capabilities for cross-modal comprehension. This involves simultaneously processing and integrating visual and textual inputs to produce meaningful, human-understandable responses. Representative tasks within this domain include image captioning, scene understanding, and visual question answering (VQA). Beyond its theoretical significance, VL understanding also serves as a foundational technology for various real-world applications, such as robotic control and autonomous driving.

However, the requirement for free-form VL understanding makes VL learning particularly challenging and resource-intensive, especially in academic research settings. Typically, researchers face limitations in two critical areas: data annotations and computational resources.

(1) Annotations serve as the fuel for modern deep-learning methods, as models require abundant annotated data to effectively generalize across diverse application scenarios. However, the long-tail nature and difficulty of fine-grained data collection makes it difficult to annotate enough data.

(2) Computational resources constitute another significant bottleneck in training VL models. Although the emergence of large-scale foundation models has greatly advanced generalized VL understanding, it simultaneously demands extensive computational power. The substantial computational requirements associated with training state-of-the-art VL models thus pose considerable barriers, particularly within academia.

Before the emergence of large foundation models (LFMs), the primary limitation in vision-and-language (VL) learning was associated with the scarcity of annotated data. To address this challenge, we developed a semi-supervised learning algorithm and validated its effectiveness on scene understanding tasks. Specifically, our approach leverages semantic hierarchies among different words to augment data, and utilize an existing model to generate annotations for unlabeled data.

Nevertheless, semi-supervised learning methods still typically require significant amounts of high-quality, in-domain annotations. To overcome this, we introduced a cross-modality prompting approach, enabling zero-shot and few-shot performance on tasks such as scene graph generation and visual grounding. This method only relies on models pre-trained with web image-text pairs.

Subsequently, the emergence of large foundation models significantly enhanced the instruction-following and conversational capabilities of VL systems. However, incorporating large language models (LLMs) also greatly increased computational demands, creating barriers for academia with limited computational resources. To mitigate this, we proposed a transfer-learning-based solution. Specifically, the visual prompt generator from previously trained VL models can be effectively transferred to new VL architectures, reducing the computational cost of training by a factor of ten.

Despite the availability of large foundation models trained on extensive web-scale image-text pairs, limitations still persist when extending models to fine-grained output formats, such as segmentation masks. To address this annotation bottleneck, we further proposed a pixel-to-embedding paradigm that unifies detection and segmentation tasks. Consequently, segmentation training benefits directly from detection training, substantially decreasing the required annotation volume.

In summary, this thesis aims to facilitate the training of powerful VL models within limited-resource contexts. Specifically, we introduce three methods to alleviate resource constraints related to data annotations and computational resources. Additionally, we put forward a proposal to extend LFM's capability to fine-grained output like segmentation masks, while maintaining moderate annotation requirement. The experimental results and revised method of the proposal will be included in the final thesis. In addition, we also include a section to discuss the future directions including the combination of current works, reasoning with small LFMs, and streaming multimodal understanding.