Reproducible configurations

[mauromorales]

After working a bit with the packages config, collector and schema. I think it would be a good idea to have reproducible configurations.

Right now, a system can have multiple sources for configuration. They can be part of one or multiple yaml files, cli flags, bootcmd attributes and urls for external configurations. All of these get merged together into a final yaml config which then gets validated. This obviously works well but can be a bit challenging when debugging or trying to understand the state in which the system is vs the one it should be.

Examples:

1. While running the installation, the system fails to fetch a certain external configuration. ATM we don't fail when we don't manage to fetch an external configuration. It doesn't matter the reason, can be because the internet connection was down or the file not being available at source. The result can be two or more different systems. One option could be to fail in case some remote config is not accessible but even then it would be nice to keep a record of what was defined and the final result. In this case for example, we should keep a record somewhere, saying that a certain source was not available at the moment of configuring. This might also be a security concern, let's say I rely on an external configuration and that configuration gets changed to point to another external source with a malitious config, we would be able to record and say config_a pointed to config_b which pointed to config_c and this is how each of them altered the system.
2. The order in which two local files are read, can change the whole configuration. For example if I have file a.yaml and file b.yaml in one of the config directories. Whatever is on b.yaml will overwrite what is on a.yaml. This is an architectural decision and not a problem, but I do think the user should be aware of this, and be able to see somewhere why things ended up the way they ended. I think it would be nice to have some way of saying, certain attribute was originally set by a.yaml and then overwritten by b.yaml. And I should be able to kind of follow step by step such changes.

My proposal would be to continue the refactoring of the config, so that such configurations can be properly audited, atomic and reproducible, but would like your input to see if you have more ideas on how to further improve this or if on the other hand you think this would be over engineering the system. So please chime in and share your thoughts :)

[jimmykarily]

I agree that it can be hard to track where settings came from. We merge configs using an external library. If we want to keep track of what overwrites what, we need a custom implementation.

On the other hand, allowing the user to generate configuration in a complex way, doesn't mean it's good practice if they do. It's rarely justified to have configuration spread in every possible source (remote config_urls, multiple files, cmdline, etc). Usually just a couple of them should be used. If the user goes down the path of complex configuration, then no auditing can save them from lost hours in debugging.

Maybe a middle ground could be to store the final applicable configuration somewhere. The user can inspect that and if it's not what they expected, they can dig deeper.
The question is, when to store this configuration. There are cases when we simply generate the whole configuration in memory just to access one key (I remember seeing such a case somewhere in the kcrypt code). Would we store the whole configuration in that case? And to go back to the auditing proposal, even if we had auditing, where would we store the information of where each setting came from?

[Itxaka]

I agree that it can be hard to track where settings came from. We merge configs using an external library. If we want to keep track of what overwrites what, we need a custom implementation.

Viper can help with this as it allows to set a fairly complex config merge order

Maybe a middle ground could be to store the final applicable configuration somewhere. The user can inspect that and if it's not what they expected, they can dig deeper.

There is currently a state yaml saved with info about the installation done by elemental here: https://github.com/kairos-io/elemental-cli/blob/main/pkg/action/install.go#L46

Maybe it could be extended to store more stuff like that and act like a source of truth, with a md5sum together to verify that it hasnt been tampered with (althougth the md5sum is kind of pointless as if you are able to modify the state yaml,you are able modify the md5sum)