Skip to content

Latest commit

 

History

History
74 lines (48 loc) · 3.38 KB

20240709_ELSA_VAJ_001.md

File metadata and controls

74 lines (48 loc) · 3.38 KB
Raw

Title of the Experiment

Connecting a GPU to a 1U server via Falcon 4005 (Standard mode)

Date and Time

07/09/2024 13:00 JST

Experimenter's Name

Shingo Hashikawa, ELSA Japan Inc.
Tomoaki Sato, VA Linux Systems Japan K.K.

Objective

Confirmation that the CDI solution can be used with GPUs even on Linux servers that do not support built-in GPUs.

Methodology

(1) Install an NVIDIA RTX A4000 GPU in the dual-width FHFL PCIe slot (1:1) of the Falcon 4005 (Standard Mode).

(2) Install host adapter on HHHL riser 1 of Dell EMC PowerEdge R450 server.

(3) Connect ports 0, 1, 2, and 3 of the Falcon 4005 to ports 0, 1, 2, and 3 of the host adapter, respectively, using four SFF-8644 cables.

(4) Start Falcon 4005.

(5) Start the Dell EMC PowerEdge R450 server.

(6) Execute the lspci command on the Linux OS to confirm that the GPU is recognized.

Materials Used

  • H3 Falcon 4005 (Standard mode) x 1
  • NVIDIA RTX A4000 GPU x 1
  • Dell EMC PowerEdge R450 x 1
    • CPU：Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz
    • MEM: 128GB
    • OS: Ubuntu 22.04.4 LTS (Linux kernel: 5.4.0-187-generic)
  • Host Adapter(PEX 88032 PCIe Gen4 Switch) x 1
  • SFF-8644 to SFF-8644 2m Copper Cable x 4

Experimental Conditions

The above verification was started by shutting down the Linux server in operation in the machine room and removing Mellanox ConnectX-6 from HHHL riser 1. HHHL riser 2c has nothing installed.

Results

An excerpt of the output of the lspci command is shown below.

 +-[0000:17]-+-00.0  Intel Corporation Device 09a2
 |           +-00.1  Intel Corporation Device 09a4
 |           +-00.2  Intel Corporation Device 09a3
 |           +-00.4  Intel Corporation Device 0998
 |           \-02.0-[18-2a]----00.0-[19-2a]--+-00.0-[1a-29]----00.0-[1b-29]----10.0-[1c-29]----00.0-[1d-29]--+-00.0-[1e-21]----00.0-[1f-21]--+-00.0-[20]--
 |                                           |                                                               |                               \-10.0-[21]--
 |                                           |                                                               +-04.0-[22-24]----00.0-[23-24]----00.0-[24]--
 |                                           |                                                               +-08.0-[25-28]----00.0-[26-28]--+-00.0-[27]--+-00.0  NVIDIA Corporation Device 24b0
 |                                           |                                                               |                               |            \-00.1  NVIDIA Corporation Device 228b
 |                                           |                                                               |                               \-10.0-[28]--
 |                                           |                                                               \-1c.0-[29]----00.0  Broadcom / LSI Device c010
 |                                           \-1c.0-[2a]----00.0  Broadcom / LSI Device c010

Analysis and Interpretation

It worked properly and no problems were detected.

Conclusions and Future Steps

GPU connectivity with the Falcon 4005 in standard mode was verified. The Falcon 4005 can be expanded from one to two target cards, and individual resources installed on the four dual-width FHFL PCIe slots can be selectively connected to hosts by purchasing an advanced mode license to connect to two hosts. This verification does not verify operation in advanced mode.