Advancing Test-Time Compute Scaling in Language Models: From Training to Inference
COM2 Level 4
Executive Classroom, COM2-04-02
Abstract:
A central challenge in test-time compute (TTC) scaling is not simply giving language models more computation, but enabling them to use it effectively. This thesis studies TTC scaling as a joint training and inference problem, spanning larger context capacity, longer reasoning trajectories, more efficient use of context, and parallel reasoning.
First, we extend usable context via continuous pretraining with an anchor-style attention mechanism that preserves compatibility with standard objectives while stably enlarging the context window. Second, we shape posttraining to elicit multi-step, deliberative reasoning so that models can productively use larger decoding budgets. Third, we modularize high-value cues with a prefix technique that decouples reusable signals from the main attention stream, reducing repetitive prompt tokens and freeing capacity for reasoning. Finally, we operationalize TTC through parallel reasoning, with multiple independent trajectories aggregated by principled ensembling.
Together, these contributions provide a unified account of how test-time computation can be expanded, used more efficiently, and organized across both sequential and parallel reasoning. They show that effective TTC scaling depends not only on allocating more computation, but also on training models and designing inference procedures that make productive use of it.