GPU Hardware: Difference between revisions

From Research Computing Center Wiki
Jump to navigation Jump to search
No edit summary
 
(29 intermediate revisions by the same user not shown)
Line 1: Line 1:




===GPU Hardware on Sapelo===
===GPU Hardware on Sapelo2===


Summary of GPU device types available on Sapelo2:
{|  width="100%" border="1"  cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | GPU model
! scope="col" | GPU devices/node
! scope="col" | CUDA cores/device
! scope="col" | Device memory
! scope="col" | GPU compute capability
! scope="col" | Minimum CUDA version supported
! scope="col" | Double-precision (fp64) support
! scope="col" | Notes
|-
|-
| H100 || 4 || 16896 || 80GB || 9.0 || 11.8 || yes || regular and buy-in
|-
| L40S || 4 || 18176 || 44GB || 8.9 || 11.8 || effectively no ||buy-in
|-
| L4 || 4 || 7424 || 24GB || 8.9 || 11.8 || effectively no || regular and buy-in
|-
| A100 || 4 || 6912|| 80GB || 8.0 || 11.0 || yes || regular and buy-in
|-
| V100 || 1 || 5120 || 32GB || 7.0|| 9.0 || yes || buy-in
|-
| V100 || 2 || 5120 || 32GB || 7.0|| 9.0 || yes || buy-in
|-
| V100 || 1 || 5120 || 16GB || 7.0|| 9.0 || yes || buy-in
|-
| V100S || 1 || 5120 || 32GB ||  7.0 || 9.0 || yes || regular and buy-in
|-
| P100 || 1 || 3584 || 16GB || 6.0 || 8.0 || yes || regular
|-
|}
====GPU resources on open-access nodes====
* 12 nodes with 4 NVIDIA H100 GPU cards each. Each of these nodes has two 32-core Intel SapphireRapids processors (64 CPU cores per node) and 1TB of RAM. The output of NVIDIA SDK deviceQuery for one such GPU card is
<pre class="gscript">
Device 0: "NVIDIA H100 80GB HBM3"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    9.0
  Total amount of global memory:                81008 MBytes (84943110144 bytes)
MapSMtoCores for SM 9.0 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 9.0 is undefined.  Default to use 128 Cores/SM
  (132) Multiprocessors, (128) CUDA Cores/MP:    16896 CUDA Cores
  GPU Max Clock rate:                            1980 MHz (1.98 GHz)
  Memory Clock rate:                            2619 Mhz
  Memory Bus Width:                              5120-bit
  L2 Cache Size:                                52428800 bytes
  Maximum Texture Dimension Size (x,y,z)        1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:              65536 bytes
  Total amount of shared memory per block:      49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                    32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:          1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                            512 bytes
  Concurrent copy and kernel execution:          Yes with 5 copy engine(s)
  Run time limit on kernels:                    No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:      Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:  0 / 85 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
</pre>
* 12 nodes with 4 NVIDIA L4 GPU cards each. Each of these nodes has two 64-core AMD EPYC Genoa processors (128 CPU cores per node) and 745GB of RAM. The output of NVIDIA SDK deviceQuery for one such GPU card is
<pre class="gscript">
Device 0: "NVIDIA L4"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    8.9
  Total amount of global memory:                22491 MBytes (23583784960 bytes)
MapSMtoCores for SM 8.9 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 8.9 is undefined.  Default to use 128 Cores/SM
  (58) Multiprocessors, (128) CUDA Cores/MP:    7424 CUDA Cores
  GPU Max Clock rate:                            2040 MHz (2.04 GHz)
  Memory Clock rate:                            6251 Mhz
  Memory Bus Width:                              192-bit
  L2 Cache Size:                                50331648 bytes
  Maximum Texture Dimension Size (x,y,z)        1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:              65536 bytes
  Total amount of shared memory per block:      49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                    32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:          1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                            512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                    No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:      Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:  0 / 33 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
</pre>
* 14 nodes with 4 NVIDIA A100 GPU cards each. Each of these nodes has two 32-core AMD EPYC 7543 processors (64 CPU cores per node) and 1TB of RAM. The output of NVIDIA SDK deviceQuery for one such GPU card is
<pre class="gscript">
Device 0: "NVIDIA A100-SXM-80GB"EPYC 7543
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    8.0
  Total amount of global memory:                81251 MBytes (85198045184 bytes)
MapSMtoCores for SM 8.0 is undefined.  Default to use 64 Cores/SM
MapSMtoCores for SM 8.0 is undefined.  Default to use 64 Cores/SM
  (108) Multiprocessors, ( 64) CUDA Cores/MP:    6912 CUDA Cores
  GPU Max Clock rate:                            1410 MHz (1.41 GHz)
  Memory Clock rate:                            1593 Mhz
  Memory Bus Width:                              5120-bit
  L2 Cache Size:                                41943040 bytes
  Maximum Texture Dimension Size (x,y,z)        1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:              65536 bytes
  Total amount of shared memory per block:      49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                    32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:          1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                            512 bytes
  Concurrent copy and kernel execution:          Yes with 5 copy engine(s)
  Run time limit on kernels:                    No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:      Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:  0 / 1 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
</pre>
* 2 nodes with AMD EPYC Rome processors (64 CPU cores and 128GB RAM per node) have one NVIDIA Tesla (Volta) V100S GPU card per node (5120 GPU cores per card). The output of NVIDIA SDK deviceQuery for one such GPU card is
<pre class="gscript">
Device 0: "Tesla V100S-PCIE-32GB"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    7.0
  Total amount of global memory:                32510 MBytes (34089730048 bytes)
  (80) Multiprocessors, ( 64) CUDA Cores/MP:    5120 CUDA Cores
  GPU Max Clock rate:                            1597 MHz (1.60 GHz)
  Memory Clock rate:                            1107 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                6291456 bytes
  Maximum Texture Dimension Size (x,y,z)        1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:              65536 bytes
  Total amount of shared memory per block:      49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                    32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:          1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                            512 bytes
  Concurrent copy and kernel execution:          Yes with 7 copy engine(s)
  Run time limit on kernels:                    No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:      Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:  0 / 196 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 1
Result = PASS
</pre>
* 2 nodes with Intel Xeon processors (32 CPU cores and 192GB RAM per node) have one NVIDIA Tesla (Pascal) P100 GPU card per node (3584 GPU cores per card). The output of NVIDIA SDK deviceQuery for one such GPU card is
<pre class="gscript">
Device 0: "Tesla P100-PCIE-16GB"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    6.0
  Total amount of global memory:                16281 MBytes (17071734784 bytes)
  (56) Multiprocessors, ( 64) CUDA Cores/MP:    3584 CUDA Cores
  GPU Max Clock rate:                            1329 MHz (1.33 GHz)
  Memory Clock rate:                            715 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                4194304 bytes
  Maximum Texture Dimension Size (x,y,z)        1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:              65536 bytes
  Total amount of shared memory per block:      49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                    32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:          1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                            512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                    No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:      Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:  0 / 59 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
</pre>
<!--
* 16 NVIDIA K40m GPU cards. These cards are installed on 2 hosts, each of which has dual 8-core Intel Xeon CPUs and 128GB of RAM; there are 8 GPU cards per host. The output of NVIDIA SDK deviceQuery for one such GPU card is
* 16 NVIDIA K40m GPU cards. These cards are installed on 2 hosts, each of which has dual 8-core Intel Xeon CPUs and 128GB of RAM; there are 8 GPU cards per host. The output of NVIDIA SDK deviceQuery for one such GPU card is


<pre class="gscript">
<pre class="gscript">
Device 0: "Tesla K40m"
Device 0: "Tesla K40m"
   CUDA Driver Version / Runtime Version          6.5 / 6.5
   CUDA Driver Version / Runtime Version          11.4 / 11.1
   CUDA Capability Major/Minor version number:    3.5
   CUDA Capability Major/Minor version number:    3.5
   Total amount of global memory:                11520 MBytes (12079136768 bytes)
   Total amount of global memory:                11441 MBytes (11996954624 bytes)
   (15) Multiprocessors, (192) CUDA Cores/MP:    2880 CUDA Cores
   (15) Multiprocessors, (192) CUDA Cores/MP:    2880 CUDA Cores
   GPU Clock rate:                               745 MHz (0.75 GHz)
   GPU Max Clock rate:                           745 MHz (0.75 GHz)
   Memory Clock rate:                            3004 Mhz
   Memory Clock rate:                            3004 Mhz
   Memory Bus Width:                              384-bit
   Memory Bus Width:                              384-bit
Line 35: Line 284:
   Device has ECC support:                        Enabled
   Device has ECC support:                        Enabled
   Device supports Unified Addressing (UVA):      Yes
   Device supports Unified Addressing (UVA):      Yes
   Device PCI Bus ID / PCI location ID:           4 / 0
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
   Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
   Compute Mode:
   Compute Mode:
     < Exclusive Process (many threads in one process is able to use ::cudaSetDevice() with this device) >
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
 
</pre>
</pre>
===GPU Hardware on zcluster===




* 32 NVIDIA Tesla (Kepler) K20Xm GPU cards (32 x 2688 = 86016 GPU cores). These cards are installed on 4 hosts each of which has dual 6-core Intel Xeon CPUs and 96GB of RAM; there are 8 GPU cards per host. The output of NVIDIA SDK deviceQuery for one such GPU card is
* 28 NVIDIA Tesla (Kepler) K20Xm GPU cards (2688 GPU cores per card). These cards are installed on 4 hosts each of which has dual 6-core Intel Xeon CPUs and 96GB of RAM; there are 7 GPU cards per host. The output of NVIDIA SDK deviceQuery for one such GPU card is
<pre class="gscript">
<pre class="gscript">
Device 0: "Tesla K20Xm"
Device 0: "Tesla K20Xm"
   CUDA Driver Version / Runtime Version          5.5 / 5.0
   CUDA Driver Version / Runtime Version          11.4 / 11.1
   CUDA Capability Major/Minor version number:    3.5
   CUDA Capability Major/Minor version number:    3.5
   Total amount of global memory:                5760 MBytes (6039339008 bytes)
   Total amount of global memory:                5700 MBytes (5976424448 bytes)
   (14) Multiprocessors x (192) CUDA Cores/MP:   2688 CUDA Cores
   (14) Multiprocessors, (192) CUDA Cores/MP:     2688 CUDA Cores
   GPU Clock rate:                               732 MHz (0.73 GHz)
   GPU Max Clock rate:                           732 MHz (0.73 GHz)
   Memory Clock rate:                            2600 Mhz
   Memory Clock rate:                            2600 Mhz
   Memory Bus Width:                              384-bit
   Memory Bus Width:                              384-bit
   L2 Cache Size:                                1572864 bytes
   L2 Cache Size:                                1572864 bytes
   Max Texture Dimension Size (x,y,z)             1D=(65536), 2D=(65536,65536), 3D=(4096,4096,4096)
   Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
   Max Layered Texture Size (dim) x layers       1D=(16384) x 2048, 2D=(16384,16384) x 2048
   Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
   Total amount of constant memory:              65536 bytes
   Total amount of constant memory:              65536 bytes
   Total amount of shared memory per block:      49152 bytes
   Total amount of shared memory per block:      49152 bytes
Line 62: Line 313:
   Maximum number of threads per multiprocessor:  2048
   Maximum number of threads per multiprocessor:  2048
   Maximum number of threads per block:          1024
   Maximum number of threads per block:          1024
   Maximum sizes of each dimension of a block:   1024 x 1024 x 64
   Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
   Maximum sizes of each dimension of a grid:     2147483647 x 65535 x 65535
   Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
   Maximum memory pitch:                          2147483647 bytes
   Maximum memory pitch:                          2147483647 bytes
   Texture alignment:                            512 bytes
   Texture alignment:                            512 bytes
Line 73: Line 324:
   Device has ECC support:                        Enabled
   Device has ECC support:                        Enabled
   Device supports Unified Addressing (UVA):      Yes
   Device supports Unified Addressing (UVA):      Yes
   Device PCI Bus ID / PCI location ID:           18 / 0
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
   Device PCI Domain ID / Bus ID / location ID:   0 / 12 / 0
  Compute Mode:
    < Exclusive Process (many threads in one process is able to use ::cudaSetDevice() with this device) >


</pre>
</pre>
-->
====GPU resources on buyin nodes====


* 10 NVIDIA Tesla (Fermi) M2070 GPU cards (9 x 448 = 4032 GPU cores). These cards are installed on 2 hosts each of which has dual 6-core Intel Xeon CPUs and 48GB of RAM; there are currently 6 GPU cards on one host and 3 on the other. The output of NVIDIA SDK deviceQuery for one such GPU card isoutput of NVIDIA SDK deviceQuery
Some research groups purchased nodes with GPU card(s) for their use.
<!--
* Two nodes with Intel Xeon processors (24 CPU cores and 128GB RAM per node) have two NVIDIA Tesla (Kepler) K80 GPU cards per node (2496 GPU cores per card). The output of NVIDIA SDK deviceQuery for one such GPU card is
<pre class="gscript">
<pre class="gscript">
Device 0: "Tesla M2070"
Device 0: "Tesla K80"
   CUDA Driver Version / Runtime Version          5.0 / 5.0
   CUDA Driver Version / Runtime Version          11.4 / 11.1
   CUDA Capability Major/Minor version number:    2.0
   CUDA Capability Major/Minor version number:    3.7
   Total amount of global memory:                6144 MBytes (6442123264 bytes)
   Total amount of global memory:                11441 MBytes (11996954624 bytes)
   (14) Multiprocessors x ( 32) CUDA Cores/MP:   448 CUDA Cores
   (13) Multiprocessors, (192) CUDA Cores/MP:     2496 CUDA Cores
   GPU Clock rate:                               1147 MHz (1.15 GHz)
   GPU Max Clock rate:                           824 MHz (0.82 GHz)
   Memory Clock rate:                            1566 Mhz
   Memory Clock rate:                            2505 Mhz
   Memory Bus Width:                              384-bit
   Memory Bus Width:                              384-bit
   L2 Cache Size:                                786432 bytes
   L2 Cache Size:                                1572864 bytes
   Max Texture Dimension Size (x,y,z)             1D=(65536), 2D=(65536,65535), 3D=(2048,2048,2048)
   Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
   Max Layered Texture Size (dim) x layers       1D=(16384) x 2048, 2D=(16384,16384) x 2048
   Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
   Total amount of constant memory:              65536 bytes
   Total amount of constant memory:              65536 bytes
   Total amount of shared memory per block:      49152 bytes
   Total amount of shared memory per block:      49152 bytes
   Total number of registers available per block: 32768
   Total number of registers available per block: 65536
   Warp size:                                    32
   Warp size:                                    32
   Maximum number of threads per multiprocessor:  1536
   Maximum number of threads per multiprocessor:  2048
   Maximum number of threads per block:          1024
   Maximum number of threads per block:          1024
   Maximum sizes of each dimension of a block:   1024 x 1024 x 64
   Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
   Maximum sizes of each dimension of a grid:     65535 x 65535 x 65535
   Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
   Maximum memory pitch:                          2147483647 bytes
   Maximum memory pitch:                          2147483647 bytes
   Texture alignment:                            512 bytes
   Texture alignment:                            512 bytes
Line 105: Line 366:
   Support host page-locked memory mapping:      Yes
   Support host page-locked memory mapping:      Yes
   Alignment requirement for Surfaces:            Yes
   Alignment requirement for Surfaces:            Yes
   Device has ECC support:                        Disabled
   Device has ECC support:                        Enabled
   Device supports Unified Addressing (UVA):      Yes
   Device supports Unified Addressing (UVA):      Yes
   Device PCI Bus ID / PCI location ID:           18 / 0
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
   Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
  Compute Mode:
    < Exclusive Process (many threads in one process is able to use ::cudaSetDevice() with this device) >
> Peer access from Tesla K80 (GPU0) -> Tesla K80 (GPU1) : Yes
> Peer access from Tesla K80 (GPU1) -> Tesla K80 (GPU0) : Yes
 
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.2, CUDA Runtime Version = 9.0, NumDevs = 2
Result = PASS


</pre>
</pre>




*One NVIDIA Tesla S1070 with four GPU cards (4 x 240 = 960 GPU cores) for programs written to use this architecture, output of NVIDIA SDK deviceQuery:
* Two nodes with Intel Xeon processors (28 CPU cores and 256GB RAM per node) have one NVIDIA Tesla (Pascal) P100 GPU card per node (3584 GPU cores per card). The output of NVIDIA SDK deviceQuery for one such GPU card is
<pre class="gscript">
Device 0: "Tesla P100-PCIE-16GB"
  CUDA Driver Version / Runtime Version          11.4 / 11.1
  CUDA Capability Major/Minor version number:    6.0
  Total amount of global memory:                16276 MBytes (17066885120 bytes)
  (56) Multiprocessors, ( 64) CUDA Cores/MP:    3584 CUDA Cores
  GPU Max Clock rate:                            1329 MHz (1.33 GHz)
  Memory Clock rate:                            715 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                4194304 bytes
  Maximum Texture Dimension Size (x,y,z)        1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:              65536 bytes
  Total amount of shared memory per block:      49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                    32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:          1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                            512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                    No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:      Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:  0 / 3 / 0
  Compute Mode:
    < Exclusive Process (many threads in one process is able to use ::cudaSetDevice() with this device) >
 
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 9.0, NumDevs = 1
Result = PASS
 
</pre>
-->
 
*Specifications for the NVIDIA L40S cards (18176 GPU cores per card). The output of NVIDIA SDK deviceQuery for these GPU cards are
<pre class="gscript">
Device 0: "NVIDIA L40S"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    8.9
  Total amount of global memory:                45468 MBytes (47677177856 bytes)
MapSMtoCores for SM 8.9 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 8.9 is undefined.  Default to use 128 Cores/SM
  (142) Multiprocessors, (128) CUDA Cores/MP:    18176 CUDA Cores
  GPU Max Clock rate:                            2520 MHz (2.52 GHz)
  Memory Clock rate:                            9001 Mhz
  Memory Bus Width:                              384-bit
  L2 Cache Size:                                100663296 bytes
  Maximum Texture Dimension Size (x,y,z)        1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:              65536 bytes
  Total amount of shared memory per block:      49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                    32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:          1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                            512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                    No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:      Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                       Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:  0 / 73 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
</pre>


* Specifications of NVIDIA Tesla (Volta) V100 and V100S GPU cards (5120 GPU cores per card). The output of NVIDIA SDK deviceQuery for these GPU cards are
<pre class="gscript">
<pre class="gscript">
Device 0: "Tesla T10 Processor"
Device 0: "Tesla V100-PCIE-16GB"
   CUDA Driver Version / Runtime Version          5.0 / 4.0
   CUDA Driver Version / Runtime Version          12.8 / 12.8
   CUDA Capability Major/Minor version number:    1.3
   CUDA Capability Major/Minor version number:    7.0
   Total amount of global memory:                4096 MBytes (4294770688 bytes)
   Total amount of global memory:                16160 MBytes (16945512448 bytes)
   (30) Multiprocessors x ( 8) CUDA Cores/MP:    240 CUDA Cores
   (80) Multiprocessors, ( 64) CUDA Cores/MP:    5120 CUDA Cores
   GPU Clock Speed:                               1.44 GHz
   GPU Max Clock rate:                           1380 MHz (1.38 GHz)
   Memory Clock rate:                            800.00 Mhz
   Memory Clock rate:                            877 Mhz
   Memory Bus Width:                              512-bit
   Memory Bus Width:                              4096-bit
   Max Texture Dimension Size (x,y,z)             1D=(8192), 2D=(65536,32768), 3D=(2048,2048,2048)
   L2 Cache Size:                                6291456 bytes
   Max Layered Texture Size (dim) x layers       1D=(8192) x 512, 2D=(8192,8192) x 512
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
   Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
   Total amount of constant memory:              65536 bytes
   Total amount of constant memory:              65536 bytes
   Total amount of shared memory per block:      16384 bytes
   Total amount of shared memory per block:      49152 bytes
   Total number of registers available per block: 16384
   Total number of registers available per block: 65536
   Warp size:                                    32
   Warp size:                                    32
   Maximum number of threads per block:          512
  Maximum number of threads per multiprocessor:  2048
   Maximum sizes of each dimension of a block:   512 x 512 x 64
   Maximum number of threads per block:          1024
   Maximum sizes of each dimension of a grid:     65535 x 65535 x 1
   Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
   Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
   Maximum memory pitch:                          2147483647 bytes
   Maximum memory pitch:                          2147483647 bytes
   Texture alignment:                            256 bytes
   Texture alignment:                            512 bytes
   Concurrent copy and execution:                 Yes with 1 copy engine(s)
   Concurrent copy and kernel execution:         Yes with 7 copy engine(s)
   Run time limit on kernels:                    No
   Run time limit on kernels:                    No
   Integrated GPU sharing Host Memory:            No
   Integrated GPU sharing Host Memory:            No
   Support host page-locked memory mapping:      Yes
   Support host page-locked memory mapping:      Yes
  Concurrent kernel execution:                  No
   Alignment requirement for Surfaces:            Yes
   Alignment requirement for Surfaces:            Yes
   Device has ECC support enabled:               No
   Device has ECC support:                       Enabled
  Device is using TCC driver mode:              No
   Device supports Unified Addressing (UVA):      Yes
   Device supports Unified Addressing (UVA):      No
  Device supports Compute Preemption:            Yes
   Device PCI Bus ID / PCI location ID:           11 / 0
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
   Device PCI Domain ID / Bus ID / location ID:   0 / 59 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >


deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 1
Result = PASS
</pre>
</pre>


*One NVIDIA C2075 GPU card (448 GPU cores), output of NVIDIA SDK deviceQuery:
<pre class="gscript">
Device 0: "Tesla V100-PCIE-32GB"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    7.0
  Total amount of global memory:                32510 MBytes (34089730048 bytes)
  (80) Multiprocessors, ( 64) CUDA Cores/MP:    5120 CUDA Cores
  GPU Max Clock rate:                            1380 MHz (1.38 GHz)
  Memory Clock rate:                            877 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                6291456 bytes
  Maximum Texture Dimension Size (x,y,z)        1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:              65536 bytes
  Total amount of shared memory per block:      49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                    32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:          1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                            512 bytes
  Concurrent copy and kernel execution:          Yes with 7 copy engine(s)
  Run time limit on kernels:                    No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:      Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:  0 / 59 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
 
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 1
Result = PASS
</pre>


<pre class="gscript">
<pre class="gscript">
Device 0: "Tesla C2075"
Device 0: "Tesla V100S-PCIE-32GB"
   CUDA Driver Version / Runtime Version          5.0 / 4.0
   CUDA Driver Version / Runtime Version          12.8 / 12.8
   CUDA Capability Major/Minor version number:    2.0
   CUDA Capability Major/Minor version number:    7.0
   Total amount of global memory:                6144 MBytes (6442123264 bytes)
   Total amount of global memory:                32510 MBytes (34089730048 bytes)
   (14) Multiprocessors x (32) CUDA Cores/MP:    448 CUDA Cores
   (80) Multiprocessors, ( 64) CUDA Cores/MP:    5120 CUDA Cores
   GPU Clock Speed:                               1.15 GHz
   GPU Max Clock rate:                           1597 MHz (1.60 GHz)
   Memory Clock rate:                            1566.00 Mhz
   Memory Clock rate:                            1107 Mhz
   Memory Bus Width:                              384-bit
   Memory Bus Width:                              4096-bit
   L2 Cache Size:                                786432 bytes
   L2 Cache Size:                                6291456 bytes
   Max Texture Dimension Size (x,y,z)             1D=(65536), 2D=(65536,65535), 3D=(2048,2048,2048)
   Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
   Max Layered Texture Size (dim) x layers       1D=(16384) x 2048, 2D=(16384,16384) x 2048
   Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
   Total amount of constant memory:              65536 bytes
   Total amount of constant memory:              65536 bytes
   Total amount of shared memory per block:      49152 bytes
   Total amount of shared memory per block:      49152 bytes
   Total number of registers available per block: 32768
   Total number of registers available per block: 65536
   Warp size:                                    32
   Warp size:                                    32
  Maximum number of threads per multiprocessor:  2048
   Maximum number of threads per block:          1024
   Maximum number of threads per block:          1024
   Maximum sizes of each dimension of a block:   1024 x 1024 x 64
   Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
   Maximum sizes of each dimension of a grid:     65535 x 65535 x 65535
   Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
   Maximum memory pitch:                          2147483647 bytes
   Maximum memory pitch:                          2147483647 bytes
   Texture alignment:                            512 bytes
   Texture alignment:                            512 bytes
   Concurrent copy and execution:                 Yes with 2 copy engine(s)
   Concurrent copy and kernel execution:         Yes with 7 copy engine(s)
   Run time limit on kernels:                    No
   Run time limit on kernels:                    No
   Integrated GPU sharing Host Memory:            No
   Integrated GPU sharing Host Memory:            No
   Support host page-locked memory mapping:      Yes
   Support host page-locked memory mapping:      Yes
  Concurrent kernel execution:                  Yes
   Alignment requirement for Surfaces:            Yes
   Alignment requirement for Surfaces:            Yes
   Device has ECC support enabled:               No
   Device has ECC support:                       Enabled
  Device is using TCC driver mode:              No
   Device supports Unified Addressing (UVA):      Yes
   Device supports Unified Addressing (UVA):      Yes
   Device PCI Bus ID / PCI location ID:           3 / 0
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
   Device PCI Domain ID / Bus ID / location ID:   0 / 196 / 0
  Compute Mode:
    < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
 
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 1
Result = PASS
</pre>
 
====Connectivity between GPU devices====
 
Each GPU device on the quad A100 nodes is connected to the other GPU devices on the same node via a 4 NVLinks. For example:
 
<pre class="gcomment">
$ nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV4 NV4 NV4 SYS 24-31 3 N/A
GPU1 NV4 X NV4 NV4 SYS 8-15 1 N/A
GPU2 NV4 NV4 X NV4 SYS 56-63 7 N/A
GPU3 NV4 NV4 NV4 X SYS 40-47 5 N/A
NIC0 SYS SYS SYS SYS X
 
Legend:
 
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
 
NIC Legend:
 
  NIC0: mlx5_0
</pre>
 
 
Each GPU device on the quad H100 nodes is connected to the other GPU devices on the same node via a 6 NVLinks. For example:
 
<pre class="gcomment">
$ nvidia-smi topo -m
    GPU0    GPU1    GPU2    GPU3    NIC0    CPU Affinity    NUMA Affinity  GPU NUMA ID
GPU0    X  NV6 NV6 NV6 NODE    0,2,4,6,8,10    0      N/A
GPU1    NV6  X  NV6 NV6 PIX 0,2,4,6,8,10    0      N/A
GPU2    NV6 NV6  X  NV6 SYS 1,3,5,7,9,11    1      N/A
GPU3    NV6 NV6 NV6  X  SYS 1,3,5,7,9,11    1      N/A
NIC0    NODE    PIX SYS SYS  X
 
Legend:
 
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
 
NIC Legend:
 
  NIC0: mlx5_0
</pre>
 
 
NVLinks are not present on the quad L4 nodes, or between GPU nodes. For example, on a quad L4 node:
<pre class="gcomment">
$ nvidia-smi topo -m
 
    GPU0    GPU1    GPU2    GPU3    NIC0    CPU Affinity    NUMA Affinity  GPU NUMA ID
GPU0    X  NODE    SYS SYS NODE    0-63    0      N/A
GPU1    NODE    X  SYS SYS NODE    0-63    0      N/A
GPU2    SYS SYS  X  NODE    SYS 64-127  1      N/A
GPU3    SYS SYS NODE    X  SYS 64-127  1      N/A
NIC0    NODE    NODE    SYS SYS  X
 
Legend:
 
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
 
NIC Legend:


  NIC0: mlx5_0
</pre>
</pre>

Latest revision as of 13:28, 29 October 2025


GPU Hardware on Sapelo2

Summary of GPU device types available on Sapelo2:

GPU model GPU devices/node CUDA cores/device Device memory GPU compute capability Minimum CUDA version supported Double-precision (fp64) support Notes
H100 4 16896 80GB 9.0 11.8 yes regular and buy-in
L40S 4 18176 44GB 8.9 11.8 effectively no buy-in
L4 4 7424 24GB 8.9 11.8 effectively no regular and buy-in
A100 4 6912 80GB 8.0 11.0 yes regular and buy-in
V100 1 5120 32GB 7.0 9.0 yes buy-in
V100 2 5120 32GB 7.0 9.0 yes buy-in
V100 1 5120 16GB 7.0 9.0 yes buy-in
V100S 1 5120 32GB 7.0 9.0 yes regular and buy-in
P100 1 3584 16GB 6.0 8.0 yes regular

GPU resources on open-access nodes

  • 12 nodes with 4 NVIDIA H100 GPU cards each. Each of these nodes has two 32-core Intel SapphireRapids processors (64 CPU cores per node) and 1TB of RAM. The output of NVIDIA SDK deviceQuery for one such GPU card is
Device 0: "NVIDIA H100 80GB HBM3"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    9.0
  Total amount of global memory:                 81008 MBytes (84943110144 bytes)
MapSMtoCores for SM 9.0 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 9.0 is undefined.  Default to use 128 Cores/SM
  (132) Multiprocessors, (128) CUDA Cores/MP:     16896 CUDA Cores
  GPU Max Clock rate:                            1980 MHz (1.98 GHz)
  Memory Clock rate:                             2619 Mhz
  Memory Bus Width:                              5120-bit
  L2 Cache Size:                                 52428800 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 5 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 85 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >


  • 12 nodes with 4 NVIDIA L4 GPU cards each. Each of these nodes has two 64-core AMD EPYC Genoa processors (128 CPU cores per node) and 745GB of RAM. The output of NVIDIA SDK deviceQuery for one such GPU card is
Device 0: "NVIDIA L4"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    8.9
  Total amount of global memory:                 22491 MBytes (23583784960 bytes)
MapSMtoCores for SM 8.9 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 8.9 is undefined.  Default to use 128 Cores/SM
  (58) Multiprocessors, (128) CUDA Cores/MP:     7424 CUDA Cores
  GPU Max Clock rate:                            2040 MHz (2.04 GHz)
  Memory Clock rate:                             6251 Mhz
  Memory Bus Width:                              192-bit
  L2 Cache Size:                                 50331648 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 33 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >


  • 14 nodes with 4 NVIDIA A100 GPU cards each. Each of these nodes has two 32-core AMD EPYC 7543 processors (64 CPU cores per node) and 1TB of RAM. The output of NVIDIA SDK deviceQuery for one such GPU card is
Device 0: "NVIDIA A100-SXM-80GB"EPYC 7543
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    8.0
  Total amount of global memory:                 81251 MBytes (85198045184 bytes)
MapSMtoCores for SM 8.0 is undefined.  Default to use 64 Cores/SM
MapSMtoCores for SM 8.0 is undefined.  Default to use 64 Cores/SM
  (108) Multiprocessors, ( 64) CUDA Cores/MP:     6912 CUDA Cores
  GPU Max Clock rate:                            1410 MHz (1.41 GHz)
  Memory Clock rate:                             1593 Mhz
  Memory Bus Width:                              5120-bit
  L2 Cache Size:                                 41943040 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 5 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
  • 2 nodes with AMD EPYC Rome processors (64 CPU cores and 128GB RAM per node) have one NVIDIA Tesla (Volta) V100S GPU card per node (5120 GPU cores per card). The output of NVIDIA SDK deviceQuery for one such GPU card is
Device 0: "Tesla V100S-PCIE-32GB"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    7.0
  Total amount of global memory:                 32510 MBytes (34089730048 bytes)
  (80) Multiprocessors, ( 64) CUDA Cores/MP:     5120 CUDA Cores
  GPU Max Clock rate:                            1597 MHz (1.60 GHz)
  Memory Clock rate:                             1107 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                 6291456 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 7 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 196 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 1
Result = PASS


  • 2 nodes with Intel Xeon processors (32 CPU cores and 192GB RAM per node) have one NVIDIA Tesla (Pascal) P100 GPU card per node (3584 GPU cores per card). The output of NVIDIA SDK deviceQuery for one such GPU card is
Device 0: "Tesla P100-PCIE-16GB"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    6.0
  Total amount of global memory:                 16281 MBytes (17071734784 bytes)
  (56) Multiprocessors, ( 64) CUDA Cores/MP:     3584 CUDA Cores
  GPU Max Clock rate:                            1329 MHz (1.33 GHz)
  Memory Clock rate:                             715 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                 4194304 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 59 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

GPU resources on buyin nodes

Some research groups purchased nodes with GPU card(s) for their use.

  • Specifications for the NVIDIA L40S cards (18176 GPU cores per card). The output of NVIDIA SDK deviceQuery for these GPU cards are
Device 0: "NVIDIA L40S"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    8.9
  Total amount of global memory:                 45468 MBytes (47677177856 bytes)
MapSMtoCores for SM 8.9 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 8.9 is undefined.  Default to use 128 Cores/SM
  (142) Multiprocessors, (128) CUDA Cores/MP:     18176 CUDA Cores
  GPU Max Clock rate:                            2520 MHz (2.52 GHz)
  Memory Clock rate:                             9001 Mhz
  Memory Bus Width:                              384-bit
  L2 Cache Size:                                 100663296 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 73 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
  • Specifications of NVIDIA Tesla (Volta) V100 and V100S GPU cards (5120 GPU cores per card). The output of NVIDIA SDK deviceQuery for these GPU cards are
Device 0: "Tesla V100-PCIE-16GB"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    7.0
  Total amount of global memory:                 16160 MBytes (16945512448 bytes)
  (80) Multiprocessors, ( 64) CUDA Cores/MP:     5120 CUDA Cores
  GPU Max Clock rate:                            1380 MHz (1.38 GHz)
  Memory Clock rate:                             877 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                 6291456 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 7 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 59 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 1
Result = PASS

Device 0: "Tesla V100-PCIE-32GB"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    7.0
  Total amount of global memory:                 32510 MBytes (34089730048 bytes)
  (80) Multiprocessors, ( 64) CUDA Cores/MP:     5120 CUDA Cores
  GPU Max Clock rate:                            1380 MHz (1.38 GHz)
  Memory Clock rate:                             877 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                 6291456 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 7 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 59 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 1
Result = PASS

Device 0: "Tesla V100S-PCIE-32GB"
  CUDA Driver Version / Runtime Version          12.8 / 12.8
  CUDA Capability Major/Minor version number:    7.0
  Total amount of global memory:                 32510 MBytes (34089730048 bytes)
  (80) Multiprocessors, ( 64) CUDA Cores/MP:     5120 CUDA Cores
  GPU Max Clock rate:                            1597 MHz (1.60 GHz)
  Memory Clock rate:                             1107 Mhz
  Memory Bus Width:                              4096-bit
  L2 Cache Size:                                 6291456 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 7 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 196 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.8, CUDA Runtime Version = 12.8, NumDevs = 1
Result = PASS

Connectivity between GPU devices

Each GPU device on the quad A100 nodes is connected to the other GPU devices on the same node via a 4 NVLinks. For example: 

$ nvidia-smi topo -m
	GPU0	GPU1	GPU2	GPU3	NIC0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV4	NV4	NV4	SYS	24-31	3		N/A
GPU1	NV4	 X 	NV4	NV4	SYS	8-15	1		N/A
GPU2	NV4	NV4	 X 	NV4	SYS	56-63	7		N/A
GPU3	NV4	NV4	NV4	 X 	SYS	40-47	5		N/A
NIC0	SYS	SYS	SYS	SYS	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0


Each GPU device on the quad H100 nodes is connected to the other GPU devices on the same node via a 6 NVLinks. For example: 

$ nvidia-smi topo -m
    GPU0    GPU1    GPU2    GPU3    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X  NV6 NV6 NV6 NODE    0,2,4,6,8,10    0       N/A
GPU1    NV6  X  NV6 NV6 PIX 0,2,4,6,8,10    0       N/A
GPU2    NV6 NV6  X  NV6 SYS 1,3,5,7,9,11    1       N/A
GPU3    NV6 NV6 NV6  X  SYS 1,3,5,7,9,11    1       N/A
NIC0    NODE    PIX SYS SYS  X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0


NVLinks are not present on the quad L4 nodes, or between GPU nodes. For example, on a quad L4 node: 

$ nvidia-smi topo -m

    GPU0    GPU1    GPU2    GPU3    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X  NODE    SYS SYS NODE    0-63    0       N/A
GPU1    NODE     X  SYS SYS NODE    0-63    0       N/A
GPU2    SYS SYS  X  NODE    SYS 64-127  1       N/A
GPU3    SYS SYS NODE     X  SYS 64-127  1       N/A
NIC0    NODE    NODE    SYS SYS  X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
Retrieved from "https://wiki.gacrc.uga.edu/index.php?title=GPU_Hardware&oldid=22574"