Investigation into the Duality of Context Prompts and Large Language Model Weights
Dr Lee Hwee Kuan, Adjunct Associate Professor, School of Computing
COM1 Level 2
SR3, COM1-02-12
Abstract:
Large Language Models (LLMs) represent one of the most significant technological advances of the twenty-first century, yet we still lack a clear understanding of how they learn and adapt. In particular, the striking phenomenon of in-context learning (ICL)—where a model appears to “learn” from a prompt without weight updates—hints at a deep link between context and model parameters. In this thesis, we investigate that link by treating prompts and weights as two sides of the same coin.
First, we show that applying ICL to a single attention layer is mathematically equivalent to performing one gradient-descent step on a specific training set. Extending this insight to multiple layers, ICL can be viewed as a greedy, layer-wise training algorithm. This perspective casts context prompts as a form of initialization and connects ICL to instruction-tuning, enriching our fundamental grasp of how LLMs process demonstrations. This furthers our fundamental understanding of ICL.
Next, we ask whether context can be permanently “installed” into a model. We introduce bias parameters that capture prompt information exactly in a linearized-attention setting, and approximately under standard vanilla attention. We present two algorithms—ICLCA (In-Context Learning Conversion Algorithm) and ICLAA (In-Context Learning Approximation Algorithm)—that embed prompt knowledge into the weights, overcoming ICL’s temporary nature and offering an interpretable, efficient update mechanism.
Finally, we leverage a context-integration framework to enhance existing fine-tuning techniques. By adding a lightweight context module, we propose Prefix-Tuning+, a modernized variant of Prefix-Tuning that addresses its limitations and achieves competitive performance with state-of-the-art parameter-efficient methods on contemporary LLMs --- something current Prefix-Tuning methods are unable to do.
Together, these contributions begin to establish a unified “duality” between prompts and weights, illuminate the mechanics of in-context learning, and pave the way for more efficient, interpretable LLM adaptation.