Efficient Data Curation And Utilization For Deep Learning
Abstract:
In the past decade, deep learning has achieved remarkable progress in the area of computer vision. Most state-of-the-art methods are trained on ultra-large-scale datasets, but the heavy training cost is hardly affordable for researchers with limited computation resources. Moreover, the cost of building the datasets was also usually beyond the capability of most researchers. Reducing the training and building effort for large-scale datasets has become urgent for broader computer vision and other deep-learning applications.
A non-negligible part of these costs comes from the redundancy of data and annotations. To address this challenge, this thesis explores different ways to enhance both training-time data efficiency and dataset efficiency (construction and utilization), thus improving the cost efficiency of large-scale datasets’ training and construction.
Firstly, the traditional random shuffling treats all samples equally important during the whole training run, which leads to redundancy in training-time computation. For already well-learned samples, repeatedly sampling them is only to keep the distribution, while these samples hardly provide much training progress. To solve this problem, we propose InfoBatch, a framework that losslessly accelerate training with unbiased dynamic data pruning. InfoBatch, as a plug-and-play and architecture-agnostic framework, consistently obtains lossless training results on classification, semantic segmentation, vision pertaining, and instruction fine-tuning tasks, saving 20% to 40% of cost with negligible overhead.
Secondly, besides the vision-only training, multi-modal training is another main-stream paradigm for vision learning. The dataset scale and quality are essential for the final performance, while the growing online data poses a challenge on maintaining a high-quality dataset. Conventional approaches takes repeated data curation with sub-optimal efficiency. To tackle this problem, we propose InfoGrowth, an efficient
online algorithm for data cleaning and selection, resulting in a growing dataset that keeps up to date with awareness of cleanliness and diversity. InfoGrowth can improve data quality/efficiency on both single-modal and multi-modal tasks, with an efficient and scalable design. Its framework makes it practical for real-world data engines.
Lastly, supervised data are always essential for many kinds of downstream tasks, and these data grow continuously. However, their annotation cost is usually much higher than the training cost in most cases. Active learning tries to reduce this cost, while usually introducing pipeline complexity and bias. To tackle this challenge, we propose Info-Coevolution, a framework that efficiently enables models and data to coevolve through online selective annotation without bias. For real-world datasets like ImageNet-1K, Info-Coevolution reduces annotation and training costs by 32% without performance drop and automatically stop with a defined criterion. Combining with semi-supervised learning, it can further save to 50% of annotation. Overall, this thesis investigates three practical methods to enhance the training/constructing efficiency of large-scale datasets. These approaches significantly reduce the cost of training and building datasets in real-world applications.