Closed-Loop Scaling: Autonomous Improvement of LLM and LVLM Reasoning
AS6 Level 5
MR6, AS6-05-10
Abstract:
The development of large language models (LLMs) and large vision language models (LVLMs) has long been guided by scaling laws. As human-annotated data approaches exhaustion and the marginal returns of pretraining diminish, however, a fundamental paradigm shift is required. This thesis proposes and formalizes automatic scaling. Rather than the traditional extrinsic scaling of parameters or data, automatic scaling refers to autonomous computational scaling, in which models leverage their own inference-time computation, self-generated feedback, and internal refinement to improve their reasoning and understanding without additional human-curated data. We formalize this as a closed-loop framework operating across three critical dimensions: inference, training, and infrastructure.
First, we investigate inference-time scaling, treating reasoning as an explicit search process rather than a linear generation task. By introducing Self-Evaluation Guided Beam Search and Event Causality via Human-centric Reasoning (ECHo), we demonstrate that models can leverage test-time computation to reason beyond their training distribution, discovering complex reasoning paths through self-critique signals and task-specific principles without additional parameter updates.
Second, we address training-time internalization. We present Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning (MCTS-IPL) and Vision-Guided Direct Preference Optimization (V-DPO), methodologies for distilling high-quality, search-discovered or multi-model co-generated trajectories back into the model's parameters. This process bridges the gap between deliberative search and efficient single-pass generation, enabling models to internalize reasoning improvements and reduce reliance on human-labeled preference data.
Third, we propose infrastructure scaling to provide the structural and contextual foundations necessary for autonomous evolution. We introduce Context-Wise Order-Agnostic Language Modeling (COrAL), a diffusion-inspired architecture that enables efficient internal iterative refinement. Complementing this, we develop interleaved vision–language chain-of-thought (VL-CoT) with visual self-reference, a paradigm that anchors multimodal reasoning in dynamic, localized visual evidence to mitigate language bias and structural hallucinations.
Together, these contributions realize a unified automatic scaling framework that transforms static models into self-improving systems. Through critical analysis, we identify the coherence–correctness gap as a systemic limitation of self-referential scaling. Specifically, self-evaluation signals at every stage of the framework, including generation, training, and evaluation, tend to favor fluent, plausible outputs over strictly correct ones. We further present MVP-Bench, a diagnostic benchmark for multi-level visual perception, whose findings reveal that current LVLMs exhibit significant deficits in high-level perceptual integration and fail to generalize to manipulated visual content. Grounded in these reflections and diagnostic evidence, we propose future directions in dynamic evaluation co-evolution, agentic scaling, and native multimodal scaling. Our results demonstrate that by scaling computation at inference and training time iteratively and grounding it through architectural and contextual innovation, foundation models can move beyond human-curated data toward autonomous, continuous improvement, while also delineating the boundaries and open challenges of this paradigm.