Why are GPUs unnecessarily idle during AI training, and what measures has Adobe taken to address this?
At NVIDIA's
Why GPUs Sit Idle: The AI Training Data Pipeline Problem
https://www.backblaze.com/blog/why-your-gpu-is-sitting-idle-the-data-pipeline-problem-no-one-talks-about/
Adobe built a massive training environment using thousands of GPUs to develop generative AI models from scratch. However, an analysis of the 'profiler output during the initial training run,' which accurately visualizes how much time each GPU spent on actual computation and how much time it spent idle, revealed that approximately two-thirds of the GPU time was spent idle (waiting for data).
Even during this idle time, there are costs associated with running the GPUs. Greenfield said, 'If you were spending $1 million on training, then $600,000 would have been wasted on GPUs that were just sitting there doing nothing.'
Adobe has identified two reasons why the GPU was idle. The first was whether the data needed for training could reach the GPU fast enough, and the second was whether the work represented by that data was evenly distributed after it arrived. Apparently, neither of these issues is specific to Adobe.
Regarding the first point, data speed, Adobe's AI training data was stored in petabyte-scale (approximately 1 million gigabytes) distributed cloud storage and needed to be constantly transmitted to thousands of GPUs via standard Ethernet. The dataset itself was enormous and diverse, with images and videos, low and high resolutions, and everything from simple formats to complex codecs being transferred simultaneously.
Traditional standard networks are built to retrieve individual files on demand and are therefore not capable of handling such persistent, parallel, petabyte-scale data transfers. Furthermore, as a safety measure, a 'checkpointing process' was performed during training, where a complete copy of the model was periodically written to storage. However, although most of these checkpoints were never needed again, the writing and reading processes consumed actual GPU time and incurred significant costs.
To address this issue, Adobe first replaced standard Ethernet with a high-performance network fabric designed to handle the petabyte-scale traffic generated by distributed AI training. This allowed them to transfer data at the speeds required by GPUs.
In addition, we've changed how checkpoints are saved. Previously, the entire model was written as one huge file, but now the model is divided into smaller fragments and saved in multiple locations at once. This has significantly reduced the time it takes to save and load checkpoints. Reconstructing the divided checkpoints takes a little time, but this isn't a problem as we've found that checkpoints are rarely needed, while the writing process, which runs continuously 24/7 on thousands of GPUs, has been greatly shortened.
The second issue, 'whether the work represented by the data is evenly distributed after it arrives,' is what Backblaze calls the 'data loader problem.'
When training an AI model on multiple GPUs, the training data is divided into chunks of equal size, and one chunk is assigned to each GPU in the cluster. However, chunks of equal size do not necessarily mean equal workloads. Some GPUs may receive simple assets that can be processed in seconds, while others receive large, complex files that take several minutes. The GPUs that finish processing quickly then become idle, waiting for the slower GPUs to catch up.
To solve the problem, Adobe explained that they stopped treating all data as the same. By building a 'proprietary balanced data loader' that distributes the work so that all GPUs complete processing at roughly the same time, they significantly reduced the wasted costs associated with waiting.
After addressing the root causes of the two issues, Adobe's GPUs are now operating at approximately 80% utilization. Considering the overhead involved in coordinating between GPUs, 80% utilization is close to the practical upper limit.
Furthermore, even a perfectly balanced data loader will remain idle if the waiting data has not yet arrived. Therefore, not only is the loader, which optimizes processing after data arrives, important, but so is the storage that determines whether data exists or not. Backblaze is promoting its object storage solution, '
Related Posts:
