Breaking Memory Walls: Efficient Memory Management for the Post-Moore's Era
COM3 Level 1
SR12, COM3 01-21
Abstract:
The relentless advancement in computing power, historically driven by Moore’s Law, has reached a critical juncture as silicon-based transistor technologies approach their physical and economic limits. In this Post-Moore’s Law era, while the demand for ever-greater computational power continues to grow—particularly in domains such as large language models (LLMs) and data-intensive applications—existing memory systems and computing architectures are increasingly constrained by fundamental bottlenecks. This thesis investigates four critical memory walls that have emerged as dominant challenges: the memory translation wall, memory capacity wall, memory bandwidth wall, and memory archival wall.
To address the memory translation wall caused by inefficiencies in page table management and TLB coherence in NUMA systems, this thesis presents Hydra, a scalable page table and TLB management system that introduces on-demand page table replication and selective TLB shootdowns to minimize cross-socket overhead and latency.
To break the memory capacity wall in GPU-based LLM inference, particularly in multi-turn conversations, this thesis proposes CachedAttention, an efficient KV cache management system. CachedAttention enables reuse of attention context across turns via an asynchronous, hierarchical caching mechanism, significantly reducing redundant computations and lowering serving costs.
To overcome the memory bandwidth wall arising from increasing parameter sizes and off-chip memory transfers in LLM inference, this work introduces IMI, a Processing-in-Memory (PIM) accelerator tailored for multi-job LLM workloads. IMI features a hybrid analog-digital computation pipeline, dynamic hardware partitioning, and slack-aware job scheduling to improve resource utilization, reduce energy costs, and enable concurrent inference workloads.
Finally, to address the memory archival wall of preserving massive LLM models, this thesis proposes Mahjong, a DNA-based storage architecture. Mahjong enhances the practicality of DNA storage by introducing error-aware bit placement, hybrid chunking, and metadata-efficient indexing strategies, offering an ultra-dense and durable solution for long-term LLM archival.
Through these contributions, this thesis provides comprehensive solutions to the critical memory management challenges of the Post-Moore's Law era, demonstrating significant improvements in performance, energy efficiency, and scalability across a range of applications.