[Feature request] Allow operator to support arbitrary YAML for agent config

[ashlineldridge]

To prevent issues like #1044 coming up as the agent evolves, it would be nice if the operator supported the ability to specify arbitrary YAML in either the form of a string or a reference to a ConfigMap that then overlaid the generated config. Is this something you would consider supporting?

[captncraig]

It is a very interesting idea, but there are a number of details that would need to be worked out.

• What is the format for defining and merging these overlays? A raw yaml blob, or some kind of jsonnet / kustomize or other dsl?
• Where do overlays live? Dynamically discovered in another crd, or mounted statically alongside the operator perhaps?
• Can you do everything you need to via agent config alone? Enabling tracing might require additional changes to pods, or new services and things like that.
• Which instances get the overlaid configs? You can potentially have multiple agent deployments per operator instance. Some will be statefulsets, others daemonsets. How would we match overlays to agent instances and ensure there aren't unintended effects on the wrong instance?

I am not saying this isn't possible, or that we'd never consider such a thing, but we'd definitely need a solid proposal that specifies all these details and more.