Revert "Change where env variable is set" #1179

Job	Run time
build (ubuntu-latest, 3.7, gcc, g++, /usr/local/lib)	12m 6s
build (ubuntu-latest, 3.7, gcc, g++, /usr/local/lib)	11m 48s
build (ubuntu-latest, 3.8, gcc, g++, /usr/local/lib)	12m 11s
build (ubuntu-latest, 3.8, gcc, g++, /usr/local/lib)	14m 3s
build (ubuntu-latest, 3.9, gcc, g++, /usr/local/lib)	12m 13s
build (ubuntu-latest, 3.9, gcc, g++, /usr/local/lib)	11m 57s
build (ubuntu-latest, 3.10, gcc, g++, /usr/local/lib)	11m 22s
build (ubuntu-latest, 3.10, gcc, g++, /usr/local/lib)	11m 22s
build (ubuntu-latest, 3.11, gcc, g++, /usr/local/lib)	11m 34s
build (ubuntu-latest, 3.11, gcc, g++, /usr/local/lib)	11m 34s
build (ubuntu-latest, 3.12, gcc, g++, /usr/local/lib)	12m 13s
build (ubuntu-latest, 3.12, gcc, g++, /usr/local/lib)	12m 32s
build (macos-latest, 3.10, cc, c++, /opt/homebrew/lib/)	11m 51s
build (macos-latest, 3.10, cc, c++, /opt/homebrew/lib/)	8m 19s
build (ubuntu-latest, 3.10, clang, clang++, /usr/local/lib/)	12m 12s
build (ubuntu-latest, 3.10, clang, clang++, /usr/local/lib/)	12m 2s
	3h 9m 19s

Provide feedback