CI: Fix cuda toolkit speed issue. #88

Job	Run time
build-shared-libs (ubuntu-latest, x86_64)	14s
build-shared-libs (ubuntu-latest, aarch64)	25s
build-shared-libs-cuda (ubuntu-latest, x86_64, 12.1.0)	2m 19s
build-shared-libs (macos-latest, x86_64)	41s
build-shared-libs-cuda (ubuntu-latest, aarch64, 12.1.0)	10m 41s
build-shared-libs-cuda (windows-latest, x86_64, 12.1.0)	17m 27s
build-shared-libs (macos-latest, aarch64)	53s
build-shared-libs (windows-latest, x86_64)	52s
build-wheels (ubuntu-latest, 3.9, x86_64)	2m 5s
build-wheels (ubuntu-latest, 3.9, aarch64)	2m 8s
build-wheels (ubuntu-latest, 3.10, x86_64)	1m 54s
build-wheels (ubuntu-latest, 3.10, aarch64)	2m 3s
build-wheels (ubuntu-latest, 3.11, x86_64)	1m 55s
build-wheels (ubuntu-latest, 3.11, aarch64)	1m 59s
build-wheels (ubuntu-latest, 3.12, x86_64)	2m 30s
build-wheels (ubuntu-latest, 3.12, aarch64)	1m 58s
build-wheels (macos-latest, 3.9, x86_64)	1m 27s
build-wheels (macos-latest, 3.9, aarch64)	1m 25s
build-wheels (macos-latest, 3.10, x86_64)	1m 29s
build-wheels (macos-latest, 3.10, aarch64)	1m 44s
build-wheels (macos-latest, 3.11, x86_64)	1m 37s
build-wheels (macos-latest, 3.11, aarch64)	1m 9s
build-wheels (macos-latest, 3.12, x86_64)	2m 31s
build-wheels (macos-latest, 3.12, aarch64)	1m 38s
build-wheels (windows-latest, 3.9, x86_64)	5m 31s
build-wheels (windows-latest, 3.10, x86_64)	3m 56s
build-wheels (windows-latest, 3.11, x86_64)	3m 35s
build-wheels (windows-latest, 3.12, x86_64)	4m 35s
publish	23s
	1h 21m 4s

Provide feedback