NVIDIA L40S GPUs - New and expensive isn't always better¶
Our xlg nodes feature NVIDIA L40S GPUs, based on the Ada Lovelace
architecture. They are a cheaper alternative to the H200 GPUs but still offer
competitive performance depending on your scientific problem. In this blog post,
we will compare the A100, H200, and L40S GPUs to help you decide which one to
use for your jobs.
Specifications¶
To start, we can look at some of the specifications of our GPUS in the table shown below.
| A100 40 GB | H200 NVL | L40S | |
|---|---|---|---|
| FP32 (TFLOPS) | 19.5 | 60.3 | 91.6 |
| TF32 (TFLOPS) | 312 | 835 | 366 |
| Memory bandwidth (TB/s) | 1.6 | 4.8 | 0.86 |
| VRAM (GB) | 40 | 141 | 48 |
| Interface | PCIe Gen 4 (64 GB/s) | PCIe Gen 5 (128 GB/s) | PCIe Gen 4 (64 GB/s) |
| NVLink | 600 GB/s | 900 GB/s | None |
Compared with the A100, the L40S offers better performance in raw compute power. This can be seen by looking at the FP32 row, which is measured in TFLOPS; this is the theoretical number of trillions of 32-bit floating-point operations per second which can be performed.
The H200 offers higher VRAM of 141 GB compared with 48 GB on the L40S. Higher VRAM means larger models or data, depending on your software, can be stored and used on the GPU. The PCIe Gen 5 interface allows for faster RAM to VRAM transfer and the 4.8 TB/s memory bandwidth allows the GPU to fetch data from VRAM at super speeds. In addition, the H200 supports NVLink, so with the correct hardware, multiple GPUs can send data to each other faster than PCIe. This makes the H200 a good candidate for machine learning problems, as the model training process involves sending training data and model updates to and from multiple GPUs at every iteration.
It gets interesting when comparing the L40S to the H200, when we look at the FP32 and TF32 rows. The L40S outperform the H200 in terms of FP32, but the H200 beats the L40S in terms of TF32. The acronym TF32 stands for TensorFloat-32. This is like FP32 but uses the higher-speed lower-precision equivalent operation instead. These operations are useful in machine learning as it trades precision for speed. For most of my scientific custom CUDA kernels, it will default to use FP32 operations. However, PyTorch will default to using TF32 operations for convolutions as mentioned in their documentation. Many neural networks are constructed using convolutions and thus use TF32 operations by default. This means that performance can depend on what libraries you use and whether you explicitly request to use FP32 or TF32 operations in your calculations.
Poisson-Ising - Single GPU for Scientific Calculations¶
We will revisit the Poisson-Ising model from a previous blog. To quickly recap, we wrote a scientific CUDA kernel that does a Gibbs sampling scheme on an image. Computationally, the image is stored in VRAM and then processed multiple times.
Figure 1: Benchmarks of different GPUs doing Gibbs sampling on the Poisson-Ising model of size 2048×2048. 1 000 samples were taken with a thinning parameter of 500, \(\lambda=0.9\) and \(\gamma=0.8\). A block dimension of 64×2 was used. The benchmarks were repeated 32 times but the variation was found to be insignificant.
Figure 1 shows the benchmarks of doing the Gibbs sampling on different GPUs. We see that the L40S beats the H200, but by a slight margin. Because I wrote the kernel myself and it heavily uses FP32 operations, the L40S will take the edge here. In addition, the algorithm does not involve a lot of data transfer between CPU and GPU; hence, we won’t make use of the higher data throughput the H200 can offer.
ImageNet - Multiple GPUs for Machine Learning¶
For our PyTorch comparison, we revisit our ImageNet blog and fit ConvNeXt onto the data using multiple GPUs.
Figure 2: Benchmarks of different GPUs fitting ConvNeXt onto the ImageNet dataset. Also shown are benchmarks when using different number of GPUs, up to four. 45 epochs with a batch size of 64 were used in the training. We used stochastic gradient descent with a decaying learning rate for optimisation. The specifications are: learning rate 0.01, momentum factor 0.9, weight decay (L2 penalty) 0.0001, period of learning rate decay 30, multiplicative factor of learning rate decay 0.1.
Figure 2 shows the benchmarks. The story is different here, as the H200 blows the L40S out of the water. This is helped by the larger memory throughput and the NVLinks, allowing training data and model updates to be quickly transferred to and between GPUs at every iteration of the training phase. Because ConvNeXt involves a lot of convolutions, PyTorch will default to using TF32 operations for them, which the H200 can process much faster than the L40S.
Conclusion¶
In conclusion, we see that the L40S and H200 can perform well in different tasks. We find that the L40S is well suited for single GPU jobs involving scientific FP32 problems, especially when we write our own CUDA kernels. But the H200 performs the best with large machine learning problems using multiple GPUs. If you're unsure which GPUs to use, I recommend investigating your software by trying out different GPUs to see which one is most suited for your problem and budget.