RFC: zephyr-dom0-xt: add possibility to use many domain configs

[GrygiriiS]

While starting working on RPI 5 Zephyr Dom0 I based on work from @xakep-amatop
'[email protected]:xakep-amatop/zephyr-dom0-xt.git' [1] which added placing and parsing of Guest domain configs from DT, but after working and playing with this for lats few week I come to the strong conclusion that this approach is not the way to move forward, because it introduces additional level of conversation from DT to C and duplicates cfg properties documentation (documented already in xen 'xl.cfg').

Lets consider adding new platform:

1. On input we get DT + requirements. To get things working we need manually fill C domain cfg from scratch.
2. On input we get xl.cfg, DT + requirement (It normally would be default use case, as it is obvious choice to get domain configuration tested with Linux before moving to Zephyr) . To get things working we need manually convert XEN xl.cfg to C domain cfg.
3. (DT approach) We still need to fill domain cfg, but in DT, which then converted to C domain cfg in Zephyr code.

In my opinion, instead of pushing code in Zephyr it better to invest in simple tool to convert xl.cfg -> C domain cfg (in C code or binary form), this way simplify adding new platforms, while keeping Zephyr code straight and simple.

Ok, above is just introduction of why I decided to switch to: xen-troops/zephyr-dom0-xt.git which now has only one working branch 'wip_dom0-xt_qemu', but it has very limited functionality. One obvious thing missed is possibility to have and use different Guest Domains configuration for different platforms.

This PR is an attempt to resolve this problem (still only one Guest domain configuration is allowed):

    Now the zephyr-dom0-xt supports only one guest domain cfg which is not
    useful if it's required to support many platforms and different
    configurations for guest domain for the same platform, like DomD and DomU.
    
    Add possibility to select which guest domain cfg source file to use at boot
    time basing on BOARD and user provided extension string which is selected
    by Kconfig options:
    
    CONFIG_USE_DEFAULT_DOM_CFG (default=y) - backward compatible behavior:
      use src/domd_cfg.c
    
    if CONFIG_USE_DEFAULT_DOM_CFG=n
      if DEFINED(CONFIG_DOM_CFG_BOARD_EXT)
        use  src/dom_cfg/dom_cfg_${BOARD}_${CONFIG_DOM_CFG_BOARD_EXT}.c
      else()
        use src/dom_cfg/dom_cfg_${BOARD}.c
    
    For example, it'd allow to have:
      dom_cfg_rpi_5_xenvm.c
      dom_cfg_rpi_5_domd.c

     Build example:
     west build -b rpi_5 -p always -S xen_dom0 zephyr-dom0-xt -- \
     -DCONFIG_USE_DEFAULT_DOM_CFG=n - DCONFIG_DOM_CFG_BOARD_EXT=\"domu\"

[1] https://github.com/xen-troops/zephyr-dom0-xt/pull/9/files

TODO List:

• Have >1 Guest Domain cfg for the same platform (might depends on xen-shell-cmd: Allow config selection for domain creation zephyr-xenlib#57)
• Untie binaries from zephyr-dom0-xt application build

[GrygiriiS]

FYI @oleksiimoisieiev @dsemenets @firscity @lorc @xakep-amatop @Deedone

[lorc]

Please ignore wip_dom0-xt_qemu branch. It was created as a early prototype to help testing zehpyr-xenlib during early development stages and it is dead end in terms of architecture. We need to design it from ground up, taking into account the following requirements:

1. Support for multiple domains. Prod-devel, for example, can launch DomD, Android VM, and generic linux guest.
2. Support for inter-domain dependencies. This means that we need to start, say, Android VM only AFTER DomD booted completely and initialized its own networking.
3. Support different HW platforms: RCar, RPi, Qualcomm, NXP....
4. Support for different "products": For example we have three RCar-based products: prod-devel, prod-aos and prod-cockpit. Each of them has different domain configuration with different hw assignment. For example, in prod-cockpit, some RCAR peripherals are passed-thru into Android VM.
5. Support for PV devices configuration
6. Support for Virtio devices configuration
7. Run-time configuration for domains. For example, in prod-devel there is a script "domd-setroot" that selects boot device for DomD based on information we get from U-Boot. I.e. in U-Boot we can tell "we want to boot whole system from SD card" or "we want to boot whole system from TFTP/NFS" and it boots without need to rebuild or recompile anything.

This is a incomplete list of features that are present in linux-based Dom0 now. Of course we don't need to implement those features all at once. But the applications should be designed with those requirements in mind. This, for instance, means that we can't have one "rpi5_domd" configuration. Because even now it is 100% clear that we will have two RPI5-based products with different HW layout.

I see it like this: this repo provides generic code, (maybe some configs, but as examples only). App is designed in a such way, that each product can provide own configuration files in some form (overlays/snippets, maybe?) to customize app behaviour for own needs.

Now, as for configuration format... We need to discuss this further, but there are some points:

1. xl.cfg format is horrific. It is inconsistent and hard to parse. Just take a look at "xl_parse.c" from xen tools and you will lose your sleep :)

2. Storing configs in manually edited C code... I dunno. I have no strong arguments against this, but I don't like this.

3. Storing config in some structured format and generating C code automatically seems as a preferred way. Again, Zephyr does exactly this. Which format to use is open for discussion. This can be xl.cfg (ugh!), it can be device tree a @xakep-amatop propoed or it can be YAML for example. We need to considers pros- and cons-

[GrygiriiS]

We need t start from something, because now there is no simple way to test xenlib.
Testing things with AOS - could be, but it too heavy wight and time consuming, plus depends on platform integration.

3. Support different HW platforms: RCar, RPi, Qualcomm, NXP....
4. Support for different "products": For example we have three RCar-based products: prod-devel, prod-aos and prod-cockpit. Each of them has different domain configuration with different hw assignment. For example, in prod-cockpit, some RCAR peripherals are passed-thru into Android VM.

This PR solves items 3 and 4 from your list. For example (CONFIG_USE_DEFAULT_DOM_CFG=n):
For 3: different HW platform:

-b rcar_h3ulcb_ca57  ----> dom_cfg_rcar_h3ulcb_ca57.c
-b rcar_spider_ca55  ----> dom_cfg_rcar_spider_ca55,.c

For 4: different HW platform + different "products":

-b rcar_h3ulcb_ca57 - DCONFIG_DOM_CFG_BOARD_EXT=\"devel\" ----> dom_cfg_rcar_h3ulcb_ca57_devel.c
-b rcar_h3ulcb_ca57 - DCONFIG_DOM_CFG_BOARD_EXT=\"aos\" ----> dom_cfg_rcar_h3ulcb_ca57_aos.c
-b rcar_h3ulcb_ca57 - DCONFIG_DOM_CFG_BOARD_EXT=\"cockpit\" ----> dom_cfg_rcar_h3ulcb_ca57_cockpit.c

Next step is to satisfy:

1. Support for multiple domains. Prod-devel, for example, can launch DomD, Android VM, and generic linux guest.

For this, It would be the way to move forward with xen-troops/zephyr-xenlib#57 , but to even start testing it with zephyr-dom0-xt - It's required to manually modify it now, for me it's not suitable at all.

Next step high priority steps:

• resolve static dependencies from binary images
• integrate xl.cfg to dom_cfg.c and switch to store dom*.cfg instead of dom_cfg*.c

Then and only then it would be possible to start doing other thing, as all devs will be on the same page and be using the same basic tool for xenlib features testing and integration.

[lorc]

For 4: different HW platform + different "products":

Just to clarify: does this imply that, say, dom_cfg_rcar_h3ulcb_ca57_devel.c will be hosted in this repo? Ideally we want to have product-specific parts in product-specific repos.

[GrygiriiS]

If build system allows - nothing prevents to add them as Pre-build step. By default we could have simple Guest cfgs (like xenvm or linux) + rpi5.

How do I see it ideally 1:

• Xen xl_"plat"yyy.cfg [+ xl"plat"_zzz.cfg] is placed in "HW prod" repo.
• Before building Zephyr: build system takes xl_yyy.cfg + xlzzz.cfg and uses the script to get cfg"plat".c, which include cfg for 2 (for example) domains
• cfg_xxx.c inserted in Zephyr to build Dom0

How do I see it Super ideally:

• Xen xl_"plat"yyy.cfg [+ xl"plat"_zzz.cfg] is placed in "HW prod" repo.
• Before building Zephyr: build system takes xl_yyy.cfg + xlzzz.cfg and uses the script to get cfg"plat".c, which include cfg for 2 (for example) domains
• Build cfg_"plat".c to get cfg_.bin
• Attach cfg_"plat".bin to the end of Dom0 zephyr.bin
• Dom0 will pick up domains cfg dynamically - even rebuild of Zephyr Dom0 is not required