rimage: add TOML preprocessing

[lyakh]

TOML configuration files for rimage are composed of multiple logical blocks, and those blocks are repeated between multiple configuration files. We use the C preprocessor to merge those blocks, to substitude any parameter values and to perform condition checking.

Note, my Python is mostly copy-pasting, so please advise any improvements.

[marc-hb]

1. Why do we need a preprocessor to just combine files? Are the requirements listed somewhere?

2. Why isn't rimage simply extended to read multiple configuration files? That would really be the simplest solution.

3. Assuming pre-processing is needed (as opposed to just combining files), why use a preprocessor literally from the 70s? There's a gazillion of more modern templating solutions in every language.

4. I'm sorry to say this but this sort of pre-processing or templating belongs to CMake, not sign.py[*]. Pretty much every build system is organized like this: a Make-like tool at the top invokes compilers and other tools that produce some output files from some input files. This is complicated and time-consuming enough with a single level but this PR adds another layer of file creation and indirection hidden from the top level CMake. This will make troubleshooting build issues harder.

4'. Generated files must be in the build directory.

[*] I'm sorry because who wants to write CMake code.

[lyakh]

1. Why do we need a preprocessor to just combine files? Are the requirements listed somewhere?

   2. Why isn't rimage simply extended to read multiple configuration files? That would really be the simplest solution.

@marc-hb as you can see in thesofproject/sof#8490 it isn't just about combining files. We had a simple PoC earlier for that that just concatenated files. We need preprocessing because (1) we want conditional module inclusion, based on Kconfig options, (2) we want to define new and to re-use existing macros from header files, (3) we want to remove the hard-coded module counter and to count modules dynamically instead.

3. Assuming pre-processing is needed (as opposed to just combining files), why use a preprocessor literally from the 70s? There's a gazillion of more modern templating solutions in every language.

See above - we're programming in a language from the 70s. I don't see why adding more tooling instead of continuing to use what we already use is better.

4. I'm sorry to say this but this sort of pre-processing or templating belongs to CMake, not sign.py[*]. Pretty much every build system is organized like this: a Make-like tool at the top invokes compilers and other tools that produce some output files from some input files. This is complicated and time-consuming enough with a single level but this PR adds another layer of file creation and indirection hidden from the top level CMake. This will make troubleshooting build issues harder.

Well, rimage is called directly from .py. If rimage were called from cmake, we could add preprocessing there too. AFAICS west sign doesn't use cmake ATM at all, does it?

4'. Generated files must be in the build directory.

Correct, that's where we put the result of toml preprocessing.

[*] I'm sorry because who wants to write CMake code.

[marc-hb]

Well, rimage is called directly from .py.

That additional layer of indirection/abstraction is unfortunate but it serves a specific purpose: rimage is not the only signing tool.

AFAICS west sign doesn't use cmake ATM at all, does it?

I hope not but west can invoke cmake sometimes in some contexts; the most obvious example is west build which can obviously invoke cmake (which can run west list... sigh)

(1) we want conditional module inclusion, based on Kconfig options, (2) we want to define new and to re-use existing macros from header files, (3) we want to remove the hard-coded module counter and to count modules dynamically instead.

All these look like very simple jobs for Python or any vaguely modern and popular scripting language. Our Kconfig implementation is entirely in Python, as are west, sign.py and most of the Zephyr build system. Python also has a TOML library.

See above - we're programming in a language from the 70s.

Python is not from the 70s, neither is CMake and TOML is 10 years old.

I don't see why adding more tooling instead of continuing to use what we already use is better.

This PR is adding new tooling; it's adding the C preprocessor in a place where it wasn't at all. Invoking the C preprocessor directly is very rare, using it to preprocess anything else than C/C++ or assembly is even more rare. This PR is an unusual and strange experiment. I suspect it's down to "let's use the only tool we know"

[lyakh]

@marc-hb have I answered your questions? Also see thesofproject/sof#8490 for SOF side motivation

[lgirdwood]

I don't see why adding more tooling instead of continuing to use what we already use is better.

This PR is adding new tooling; it's adding the C preprocessor in a place where it wasn't at all. Invoking the C preprocessor directly is very rare, using it to preprocess anything else than C/C++ or assembly is even more rare. This PR is an unusual and strange experiment. I suspect it's down to "let's use the only tool we know"

@marc-hb SOF has been using TOML for the last 4 years and there are no plans to change. The pre-processor is the right tool today for this use case as all the SOF developers understand pre-processing directives to the level required. There is a tight coupling between the SOF C code/Kconfig contents and the toml data at build time (i.e. we need to pre-process both with the same rules and macros for all C/toml files to make sure everything is aligned).

[marc-hb]

Most of the code looks ok but there are some big UI and backwards compatibility issues with the cmake_toml = target + '.toml.h' logic.

First, this introduces a "flag day" where both SOF and Zephyr have to be changed at the same time. It will block git bisect and similar things. That's not necessary.

The new, hard dependency on .toml.h will also make it harder to test and debug pure sign.py issues in isolation.

Most importantly, this PR missed the fact that the -c option can come from multiple places: see lines 559. So this PR demands .toml.h and pre-processing when -c comes from some place but demands the older and plain .toml when -c comes from other places. That inconsistency is not OK

All these issues are hopefully easy to address:

1. move the logic later, AFTER all -c origin have been considered.
2. something like:

  if exists(.toml.h):
      pre-process
      use build/zephyr/.toml
  else if exists(.toml)
      use src/.toml
  else
      raise exception(missing .toml)

[marc-hb]

Move this at the top to avoid the "lost" and barely visible else clause.

conf_path_cmd = []
if conf_dir:
    ...
    conf_path_cmd = ['-c', output_toml]

[marc-hb]

All these issues are hopefully easy to address:

After more thought, it may not be that easy.

BTW this PR seems currently incompatible with xtensa-build-zephyr.py which is what SOF CI and most SOF developers (but apparently not @lyakh ?) use to build SOF.

[marc-hb]

BTW this PR seems currently incompatible with xtensa-build-zephyr.py which is what SOF CI and most SOF developers (but apparently not @lyakh ?) use to build SOF.

Right now the combination of these two PRs builds neither TGL nor MTL.

• TGL doesn't build because xtensa-build-zephyr.py passes "-c sof/rimage/config/tgl.toml" and that takes precedence over conf_dir which is mostly meant as a fallback. In other words the output of this new code is ignored. sof/rimage/config/tgl.toml has been split and removed and is not found => fail

• MTL fails even earlier because even when "-c sof/rimage/config/mtl.toml" is passed and takes precedence... but only AFTER this new code tries to preprocess. Which fails because mtl.toml has not been split yet and mtl.toml.h is not found yet.

TL;DR: rimage configuration is just quite complicated already even before this split.

There are some options, I'm looking at them and experimenting.

[marc-hb]

Candidate replacement:

• rimage: pre-process .toml files with cc -E #66420
• xtensa-build-zephyr.py: don't pass -c $plat.toml to rimage  thesofproject/sof#8602

Both are required for this to work but they can be merged in any order.

[marc-hb]

@lyakh close?