Enhancing Computational Resource Utilization in Large-Scale Neural Networks
Abstract:
In recent years, deep learning has consistently set new records across various tasks, such as computer vision and natural language processing. A common approach in the community to build powerful models involves scaling up both model size and dataset size. While this enhances model capabilities, it also introduces significant training and inference costs.
One major source of these costs is that large-scale models may not fully utilize computational resources due to memory limitations or inefficient reusage of pretrained models. To address this challenge, this proposal explores different methods to enhance computational resource utilization for large-scale neural networks.
Firstly, training large-scale models requires parallelism across multiple computa- tional nodes. Data parallelism allows for the use of large batch sizes to maximize each machine’s power. However, training performance often deteriorates with large batch sizes. To solve this dilemma, we propose CowClip, an optimizer designed to stabilize training behavior specifically for large batch training. CowClip achieves lossless performance with a 128x batch size and a 72x speedup on click-through rate prediction tasks.
Additionally, large-scale models present significant challenges for inference. For example, predicting a single token in a large language model requires a forward pass, and without batching, GPU utilization is inefficient. Different response lengths within a batch can also lead to inefficiencies, as shorter generations must wait for longer ones to complete. To address this, we propose Sequence-Scheduling, which forms batches based on predicted lengths, based on a trained response length perception module. It significantly speeds up inference throughput.
Lastly, training large-scale pretrained models demands substantial resources. To mitigate this, we aim to enable lifelong learning for pretrained models. This approach allows for efficient knowledge injection and updates without retraining the entire model, thereby reducing computational resources and facilitating post-training for downstream tasks.