compose: existing resource support

[weikanglim]

Background

This issue tracks the ability to reference existing resources in azure.yaml.

For example, the schema could look something like:

resources:
  existing-db:
    id: <Azure Resource ID>

This enhancement builds on top of the idea that azure.yaml is representing the app-centric view of the infrastructure resources. Similar to a SQL view, resources can either be created (materialized in SQL terms) or existing (queried from in SQL terms).

Scenarios

When this type of reference metadata is present, scenarios we would want to enable for overall consistency of the system:

1. use binding relationships works with existing resources. We could imagine that environment variables are established in the same way a created resource does. This would require updating the infra synthesis to account for existing declarations.
2. Command gestures like show works with existing resources. This change would be a minimal addition of the currently established design, that each resource maps to an AZURE_RESOURCE_xx_ID upon provisioning. See compose: show <resource> #4534 for details.

[timngmsft]

Have you given thought to separating the resource ids from the yaml file? There’s two reasons I’ve been thinking about:

1. Maybe resource ids shouldn’t be checked into a repository
2. If you end up separating it, it’s possible to support multiple “environments”?

For example, you could put the resource ids in “dev.env” and “prod.env” and azure.yaml can reference the IDs in these environment files?

What do you think of that kind of approach?

[weikanglim]

100% agree that multiple environments scenarios should always be supported.

Current established paradigms:

There are many ways we can do that, currently:

1. The current, already supported approach. Define in azure.yaml, reference the environment variable:

resources:
  existing-db:
    id: ${ACTUAL_ID_KEY}

2. Define environment variable directly: ${AZURE_RESOURCE_xx_ID} -- This could serve as an override.

Future Improvements

Some alternative improvements over the long run:

1. More explicit parameterization with ${env.someId}
2. Some other wild ideas: Explicit configuration files that are merged together, this are now source-control managed. Perhaps you could think of:
   • azure.yaml
   • azure.prod.yaml - With prod specific configuration

I personally see that 1 and 2 become increasing cumbersome over the long run, and perhaps at some breaking point, separate configuration files are more maintainable.

I do very much see the need to have a system of this essence very clearly in-place and well-documented as a pattern.

[timngmsft]

I think using the environment variable is actually a fine approach; users can then supply the right system environment variables as they need.

The "azure.yaml" and "azure.prod.yaml" is interesting - have you heard of customers needing different settings in dev/prod? I remember talking to some customers about it and it seems to resolve around configuration - for example, the dev environment has smaller VMs, lower SLAs, etc. This would be possible using the environment approach as well correct?