GCC Cross Compiler build scripts

The repository is part of the Compiler Explorer project. It builds the docker images used to build the various GCC cross-compilers used on the site.

How to add a new version for some/all cross compilers

The script check_and_update_conf.py can be used to automate some work:

• update the various configurations for our nodes on AWS (i.e. all the *.amazon.properties in the compiler-explorer repository).
• while doing so, it checks the mapping between the languages enabled in the crosstool-ng config file and the actually installed compiler.
• generate a simple script to smoketest the newly added compilers using the API

Beware that you should not trust blindly everything the script does. You should probably still try manually (or at least some random combination if testing everything is too much).

The script is invoked for a given target arch and will do its magic from this point. Here's a sample execution for adding GCC 12.2.0 for ARM64:

$ ./check_and_update_conf.py -a arm64 --version 12.2.0\
   --guess-previous\
   --config-dir ~/git/compiler-explorer/compiler-explorer/etc/config/\
   --inplace \
   --config-todo fixups.txt \
   --summary summary.txt \
   --create-api-tests tests.sh

This will read/modify (--inplace) needed config files located in ~/git/compiler-explorer/compiler-explorer/etc/config. The script tries to be smart and find a previous matching compiler for all languages for the target, but this may not be possible (e.g. the naming pattern is different or there is really no previous one). In this case, the script still produces some configuration that you need to add manually. It in the fixups.txt (as specified by --config-todo fixups.txt):

Please add the following in /bla/git/compiler-explorer/compiler-explorer/etc/config/c++.amazon.properties:
8<---8<--- BEGIN ---8<---8<---

compiler.rv32-cgcc1220.exe=/opt/compiler-explorer/riscv32/gcc-12.2.0/riscv32-unknown-linux-gnu/bin/riscv32-unknown-linux-gnu-g++
compiler.rv32-cgcc1220.semver=12.2.0
compiler.rv32-cgcc1220.objdumper=/opt/compiler-explorer/riscv32/gcc-12.2.0/riscv32-unknown-linux-gnu/bin/riscv32-unknown-linux-gnu-objdump
compiler.rv32-cgcc1220.demangler=/opt/compiler-explorer/riscv32/gcc-12.2.0/riscv32-unknown-linux-gnu/bin/riscv32-unknown-linux-gnu-c++filt
compiler.rv32-cgcc1220.name=riscv32 12.2.0

8<---8<--- END ---8<---8<---

The --summary summary.txt instructs the script to create a small summary of what happened and what you still need to do by hand:

ALREADY EXISTS: riscv32 12.2.0 D
ALREADY EXISTS: riscv32 12.2.0 FORTRAN
MANUAL FIXUP NEEDED: riscv32 12.2.0 CXX
ALREADY EXISTS: riscv32 12.2.0 C
ALREADY EXISTS: riscv64 12.2.0 ADA
ALREADY EXISTS: riscv64 12.2.0 D
ALREADY EXISTS: riscv64 12.2.0 FORTRAN
MANUAL FIXUP NEEDED: riscv64 12.2.0 CXX
ALREADY EXISTS: riscv64 12.2.0 GO
ALREADY EXISTS: riscv64 12.2.0 C
ALREADY EXISTS: mipsel 12.2.0 D
ALREADY EXISTS: mipsel 12.2.0 FORTRAN
ALREADY EXISTS: mipsel 12.2.0 CXX
ALREADY EXISTS: mipsel 12.2.0 GO
ALREADY EXISTS: mipsel 12.2.0 C
ALREADY EXISTS: mips64el 12.2.0 FORTRAN
ALREADY EXISTS: mips64el 12.2.0 CXX
ALREADY EXISTS: mips64el 12.2.0 GO
ALREADY EXISTS: mips64el 12.2.0 C

It can also create a shell script (it has been shellchecked, but would not qualify for best shell-style) to tests that all newly added compilers are behaving. You still need to check if the results are expected as the script can't really infer all prop values (in particular for binarySupports): --create-api-tests tests.sh --api-test-host http://localhost:10240. The test scripts also contains tests that are expected to FAIL in order to test the test harness. These tests are clearly identified in the result file (test.result).

The test script doesn't take any argumuent:

$ bash tests.sh

And the test.result looks like:

mipsel D gdcmipsel1220 ASM-------------- [OK]
mipsel D gdcmipsel1220 ASM+BINARY------- [OK]
mipsel FORTRAN fmipselg1220 ASM--------- [OK]
mipsel FORTRAN fmipselg1220 ASM+BINARY-- [SKIPPED (not supported)]
mipsel CXX mipselg1220 ASM-------------- [OK]
mipsel CXX mipselg1220 ASM+BINARY------- [OK]
mipsel GO gccgomipsel1220 ASM----------- [OK]
mipsel GO gccgomipsel1220 ASM+BINARY---- [SKIPPED (not supported)]
mipsel C cmipselg1220 ASM--------------- [OK]
mipsel C cmipselg1220 ASM+BINARY-------- [OK]


#### Fake tests, they should FAIL or be SKIPPED, but never PASS

mipsel ADA gdcmipsel1220 ASM------------ [FAIL]
mipsel ADA gdcmipsel1220 ASM+BINARY----- [FAIL]
#### End of fake tests

mips64el FORTRAN fmips64elg1220 ASM----- [OK]
mips64el FORTRAN fmips64elg1220 ASM+BINARY [SKIPPED (not supported)]
mips64el CXX mips64elg1220 ASM---------- [OK]
mips64el CXX mips64elg1220 ASM+BINARY--- [OK]
mips64el GO gccgomips64el1220 ASM------- [OK]

When you need to update several targets at once, you can use the following sample command. You currently need to clear some output files as they are appended by the several invocations.

rm -f test.result tests.sh summary.txt test.result fixups.txt;\
  for i in arm arm64 avr mips mips64 msp430 powerpc powerpc64 powerpc64le s390x riscv32 riscv64 mipsel mips64el;
    do
      ./check_and_update_conf.py -a $i --version 12.2.0 --guess-previous \
         --config-dir ~/git/compiler-explorer/compiler-explorer/etc/config/\
         --inplace  --config-todo fixups.txt --summary summary.txt --create-api-tests tests.sh
    done