[Issue]: Bad parsing of nvcc command line arguments by hipcc

[neworderofjamie]

Problem Description

I'm trying to port some CUDA code to HIP and the --compiler-options "-fPIC"command line arguments seems to be being parsed incorrectly on the nvidia platform and not being passed through to nvcc

Operating System

Ubuntu 20.04.6 LTS (Focal Fossa)"

CPU

Intel(R) Xeon(R) Gold 6134 CPU

GPU

2*NVIDIA RTX A5000

ROCm Version

ROCm 6.2.4

ROCm Component

HIPCC

Steps to Reproduce

1. Run command like /opt/rocm/bin/hipcc -dc -x cu -arch sm_86 -I"/opt/rocm/include" -std=c++11 --compiler-options -fPIC file.cc where file.cc is a HIP source
2. Observe that hipcc tries to execute /usr/local/cuda/bin/nvcc -Wno-deprecated-gpu-targets -lcuda -lcudart -L/usr/local/cuda/lib64 -dc -x cu -arch sm_86 -I/opt/rocm/include -std=c++11 --compiler-options file.cc i.e. compiler-options is parsed incorrectly

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

I can work around this by setting NVCC_APPEND_FLAGS="--compiler-options -fPIC" but this is annoying

[schung-amd]

Hi @neworderofjamie, you should be using -Xcompiler instead of --compiler-options to pass compiler options to nvcc through hipcc; see

llvm-project/amd/hipcc/bin/hipcc.pl

Lines 354 to 361 in f9dc83d

	# nvcc does not handle standard compiler options properly
	# This can prevent hipcc being used as standard CXX/C Compiler
	# To fix this we need to pass -Xcompiler for options
	if (($arg eq '-fPIC' or $arg =~ '-Wl,') and $HIP_COMPILER eq 'nvcc')
	{
	$HIPCXXFLAGS .= " -Xcompiler ".$arg;
	$swallowArg = 1;
	}

This part of the script appears to specifically handle -fPIC as well, so you should be able to just pass -fPIC.

[neworderofjamie]

Thanks for your response but -Xcompiler seems to have the same parsing bug as --compiler-options i.e. if I run:

/opt/rocm/bin/hipcc -dc -x cu -arch sm_86  -I"/opt/rocm/include" -std=c++11 -Xcompiler -fPIC files.cc

nvcc is incorrectly invoked with:

ailed to execute:/usr/local/cuda/bin/nvcc  -Wno-deprecated-gpu-targets -lcuda -lcudart -L/usr/local/cuda/lib64  -dc -x cu -arch sm_86 -I/opt/rocm/include -std=c++11 -Xcompiler file.cc

Using plain -fPIC seems to launch nvcc correctly but doesn't seem to actually set the flag. If I run:

HIPCC_VERBOSE=1 /opt/rocm/bin/hipcc -dc -x cu -fPIC -arch sm_86  -I"/opt/rocm/include" -std=c++11 file.cc

you can that the -fPIC is being stripped:

hipcc-cmd: /usr/local/cuda/bin/nvcc  -Wno-deprecated-gpu-targets -lcuda -lcudart -L/usr/local/cuda/lib64  -dc -x cu -arch sm_86 -I/opt/rocm/include -std=c++11 neuronUpdate.cc

[neworderofjamie]

Also, by default my computer is using the binary rather than perl hipcc. However, if I force the perl version, I think the problem is with -x cu not being parsed correctly, $hasCXX not being set and thus $HIPCXXFLAGS not being added to the command line. hipcc seems to be barely documented (--help just forwards to nvcc) so I'm not really sure if this if -x cu is even a supported command line argument.

[schung-amd]

Thanks for checking it out! The binary and perl script should have the same logic with regards to this. I think you're on the right track, and agree that this is due to poor documentation:

I'm not really sure if this if -x cu is even a supported command line argument.

It should be, but the scripts are actually configured to add -x cu by default on the nvidia platform for several filetypes:

llvm-project/amd/hipcc/bin/hipcc.pl

Lines 417 to 422 in fba2930

	} elsif (($arg =~ /\.cpp$/) or ($arg =~ /\.cxx$/) or ($arg =~ /\.cc$/) or ($arg =~ /\.C$/)) {
	$needCXXFLAGS = 1;
	if ($HIP_COMPILE_CXX_AS_HIP eq '0' or $HIP_PLATFORM ne "amd" or $hasOMPTargets eq 1) {
	$hasCXX = 1;
	if ($HIP_PLATFORM eq "nvidia") {
	$toolArgs .= " -x cu";

and passing -x cu explicitly seems to cause issues. On my end,

hipcc -fPIC test.cc

translates to

/usr/local/cuda/bin/nvcc  -Wno-deprecated-gpu-targets  -isystem /usr/local/cuda/include -isystem "/opt/rocm-6.3.0/include" -Xcompiler -fPIC -x cu  -Wno-deprecated-gpu-targets -lcuda -lcudart -L/usr/local/cuda/lib64  test.cc

which has the expected -fPIC flag, while

hipcc -x cu -fPIC test.cc

translates to

/usr/local/cuda/bin/nvcc  -Wno-deprecated-gpu-targets -lcuda -lcudart -L/usr/local/cuda/lib64  -x cu test.cc

which does not seem to set the proper flags as you have observed. I see an old internal issue pertaining to this but can't find any mentions of this in documentation.

Hopefully this resolves the issue on your end. I'll take a look to see where we can add documentation around this regardless, because there is no indication as far as I can see that adding -x cu would break anything.

[neworderofjamie]

Thank you so much - getting rid of the -x cu seems to fix the issue