Locality Driven Optimization for Resource-Efficient Machine Learning Inference
COM2 Level 4
Executive Classroom, COM2-04-02
Abstract:
Machine learning (ML) inference spans a wide range of deployment platforms, from high-performance servers to memory- and energy-constrained edge devices, each imposing unique constraints on compute, memory, and energy efficiency. While traditional layered optimizations have improved performance across the model, software, and hardware stack, they often fail to capture dynamic and cross-layer forms of locality that are critical for efficient execution. This thesis examines platform-specific strategies for exposing and exploiting such locality, aligning model behavior with system capabilities to enhance inference efficiency. At the server scale, we introduce DAOP, a runtime-aware inference engine for Mixture-of-Experts (MoE) models that exploits sequence-level activation patterns to guide parallel expert execution across GPUs and CPUs. DAOP dynamically predicts expert usage and precomputes experts on the CPU to mitigate GPU memory bottlenecks. To accelerate CoT-style reasoning, SAEM introduces stage-aware expert caching and kernel-level optimizations, improving efficiency under long-context, autoregressive decoding. On mobile systems, Para-Pipe exploits structural locality in computation graphs through hierarchical operator mapping, enabling flexible trade-offs between latency, throughput, and energy consumption on heterogeneous architectures. For ultra-low-power TinyML systems, we provide a full-stack performance and power analysis that identifies key inefficiencies across model, library, and hardware layers. Collectively, these contributions demonstrate that locality-aware design enables scalable and efficient ML inference across diverse platforms.